The "^'' glyph in the TeXBook

Question

TeX, plain TeX, within the source of the Texbook of Don Knuth, there is use of ^ in different command sequences, and in text bites.

Is there a special meaning to ^ in TeX primitives, in plain TeX, is it a part of a macro defined in Knuth's proprietary input files to the TeXbook?

The output seems not to betray the intended use.

Answer 1

The manmac format used for processing the TeXbook defines ^ in a very special way.

Under the standard setup, ^ alone is the marker for superscripts and also has a very peculiar behavior when followed by itself, see What is the role of an unescaped circumflex or hat character " ^ "?

However, manmac.tex makes ^ into an active character and Knuth uses it for indexing:

% Indexing macros
\newif\ifproofmode
\proofmodetrue % this should be false when making camera-ready copy
\newwrite\inx
\immediate\openout\inx=index % file for index reminders
\newif\ifsilent
\def\specialhat{\ifmmode\def\next{^}\else\let\next=\beginxref\fi\next}
\def\beginxref{\futurelet\next\beginxrefswitch}
\def\beginxrefswitch{\ifx\next\specialhat\let\next=\silentxref
  \else\silentfalse\let\next=\xref\fi \next}
\catcode`\^=\active \let ^=\specialhat
\def\silentxref^{\silenttrue\xref}

Let's look at some calls from the source of the TeXbook.

Another noteworthy characteristic of this manual is that it doesn't
always tell the ^{truth}.

Let's keep the name \TeX\
for the language described here, since it is so much better, and since
it is not going to change any more. ^^{TeX78}

medium that doesn't allow lowering of the `E', is to type `^|TeX|'. Then

The solution is to type \thinspace|'\thinspace''|, which
produces '\thinspace'' as desired.^^|\thinspace|

Snippet 1 has a word that is both indexed and printed; snippet 2 has a word that is indexed but not printed. Snippets 3 and 4 are similar, but the term is stored verbatim in the index auxiliary file.

Note that in the TeXbook, |...| is always used as a shorthand for printing a term verbatim.

The character ^ is made active and its meaning is made identical to \specialhat.

If the character is found in math mode, a standard category code ^ is delivered to the main input stream, thus enabling usage of superscripts with the standard notation.

In text mode it is different, because \beginxref is delivered.

The \beginxref macro checks the next token with \futurelet. If the next token is ^, \silentxref is delivered, otherwise \ifsilent is set to false and \xref is delivered.

As you see, \silentxref gobbles the next ^ and does \silenttrue\xref.

Describing \xref would be very long and not relevant to the description you were looking for.

A subtle point to be noted. The ^ in the replacement text for \specialhat appears before the change in category code, so it is the normal “superscript” character (tokens in the replacement text are “frozen” at definition time). The ^ in the parameter text of \silentxref appears after the category code change, so it should have category code 13 when \silentxref is expanded (which is normally already ensured by the fact that \silentxref has been called).

---

Just to play around, here's an expl3 version of the code (just the macros, not the setup for writing to the auxiliary index file)

\bool_new:N \l_manmac_silent_bool
\cs_new_protected:Nn \manmac_specialhat:
 {
  \mode_if_math:TF { ^ } { \manmac_beginxref: }
 }
\cs_new_protected:Nn \manmac_beginxref:
 {
  \peek_charcode_remove:NTF ^
   {
    \bool_set_true:N \l_manmac_silent_bool \manmac_xref:
   }
   {
    \bool_set_false:N \l_manmac_silent_bool \manmac_xref:
   }
 }
\char_set_active_eq:NN ^ \manmac_specialhat:
\char_set_catcode_active:N ^

I believe this code makes clearer the actions described above, because details like the several assignments to \next can be avoided.

Here's a \let\next free version of the same macros

\newif\ifsilent
\def\specialhat{%
  \ifmmode
    \expandafter^%
  \else
    \expandafter\beginxref
  \fi
}
\def\beginxref{\futurelet\next\beginxrefswitch}
\def\beginxrefswitch{%
  \ifx\next\specialhat
    \expandafter\silentxref
  \else
    \silentfalse\expandafter\xref
  \fi
}
\catcode`\^=\active \let ^=\specialhat
\def\silentxref^{\silenttrue\xref}

Answer 2

If you are referring to the use of ^ like in ^^{TeX78} and ^^|\copyright|, here's the relevant part from manmac.tex

% Indexing macros
\newif\ifproofmode
\proofmodetrue % this should be false when making camera-ready copy
\newwrite\inx
\immediate\openout\inx=index % file for index reminders
\newif\ifsilent
\def\specialhat{\ifmmode\def\next{^}\else\let\next=\beginxref\fi\next}
\def\beginxref{\futurelet\next\beginxrefswitch}
\def\beginxrefswitch{\ifx\next\specialhat\let\next=\silentxref
  \else\silentfalse\let\next=\xref\fi \next}
\catcode`\^=\active \let ^=\specialhat
\def\silentxref^{\silenttrue\xref}

Basically it defines \specialhat and then does \let^=\specialhat. It's his way of doing a clean reference system to the index (he seemed to like ^^{foo} notation I guess, rather than, say \index{..} or \index|..|).

Then he defines \xref which is the actual function. ^ is just the “public interface” to \xref.

\def\marginstyle{\vrule height6pt depth2pt width\z@ \sevenrm}

\chardef\bslash=`\\
\def\xref{\futurelet\next\xrefswitch}
\def\xrefswitch{\begingroup
  \ifx\next|\aftergroup\vxref % case 1 or 2, |arg| or |\arg|
  \else\ifx\next\<\aftergroup\anglexref % case 3, "\<arg>" means angle brackets
    \else\aftergroup\normalxref \fi\fi\endgroup} % case 0, "{arg}"
\def\vxref|{\catcode`\\=\active \futurelet\next\vxrefswitch}
\def\vxrefswitch#1|{\catcode`\\=0
  \ifx\next\empty\def\xreftype{2}%
    \def\next{{\tt\bslash\text}}% type 2, |\arg|
  \else\def\xreftype{1}\def\next{{\tt\text}}\fi % type 1, |arg|
  \edef\text{#1}\makexref}
{\catcode`\|=0 \catcode`\\=\active |gdef\{}}
\def\anglexref\<#1>{\def\xreftype{3}\def\text{#1}%
  \def\next{\<\text>}\makexref}
\def\normalxref#1{\def\xreftype{0}\def\text{#1}\let\next=\text\makexref}
\def\makexref{\ifproofmode\insert\margin{\hbox{\marginstyle\text}}%
   \xdef\writeit{\write\inx{\text\space!\xreftype\space
     \noexpand\number\pageno.}}\writeit
   \else\ifhmode\kern\z@\fi\fi
  \ifsilent\ignorespaces\else\next\fi}
% the \insert (which is done in proofmode only) suppresses hyphenation,
% so the \kern\z@ is put in to give the same effect in non-proofmode.

I'm not an expert, others might give better details and reasons for this.

Answer 3

The ^ character is used either to indicate the switch to superscript typesetting or it is a character replacement method (better shifting method),e.g.

^^+ will be replaced by the character k, since k and + differ by 64 positions in the ordinary ASCII code, it's possible to say

\vs^^+ip 8cm instead of \vskip 8cm

Identically, ^^k will replaced by + then, but this hard to read, in my point of view.

The ^^ code is called character internal representation.

Here's a strange way of saying Hello World (compile with TeX or pdfTeX)

H^^%^^,^^,^^/  W^^/^^2^^,^^$
\^^"^^9^^%