Trying to understand a line in latex.ltx (regarding \string, \escapechar and \csname in the definition of \newif)

Question

I'm trying to understand how exactly TeX parses documents, and with respect to that, the definition of \newif confuses me quite a lot (in latex.ltx, line 929ff):

\def\newif#1{%
  \count@\escapechar \escapechar\m@ne
    \let#1\iffalse
    \@if#1\iftrue
    \@if#1\iffalse
  \escapechar\count@}
\def\@if#1#2{%
  \expandafter\def\csname\expandafter\@gobbletwo\string#1%
      \expandafter\@gobbletwo\string#2\endcsname
          {\let#1#2}}

I understand what this definition does (i.e. \newif\iffoo introduces three macros \iffoo, \foofalse and \footrue), but I don't understand why it works. Here's how I understand this code:

• \newif\iffoo first stores the current escape character in \count@ and temporarily redefines it to -1. This is probably the crux of the issue, but it's unclear to me how exactly that affects further parsing.
• We define \iffoo to be \iffalse. Apparently the redefining of \escapechar has no effect here yet. So far this makes sense, since (I guess) the "definition" of \newif and the argument \iffoo have already been tokenized before the \escapechar-stuff is expanded(?).
• Now the weird stuff that \@if does comes into play. We do \@if\iffoo\iftrue, which does the \csname-stuff. And here's where I don't understand how this works. I read this as:
  • \expandafter\def\csname\expandafter\@gobbletwo\string#1\expandafter\@gobbletwo\string#2\endcsname expands to
  • \def[\csname\@gobbletwo[\string\iffoo]\@gobbletwo[\string\iftrue]\endcsname] (where by the square brackets I mean the evaluation of).
  • According to the TeXbook, \string turns a control sequence into its constituent characters as individual tokens, including the escape character, all with category code 12. So \string\iffoo should turn into $\backslash_{12}i_{12}f_{12}f_{12}o_{12}o_{12}$. But this would result in \csname ffooftrue\endcsname rather than \csname footrue\endcsname.

Quote TeX book:

Although control sequences are treated as single objects, TEX does provide a way to break them into lists of character tokens: If you write \string\cs, where \csis any control sequence, you get the list of characters for that control sequence’s name. For example, \string\TeX produces four tokens: $\backslash_{12},T_{12},e_{12},X_{12}$. Each character in this token list automatically gets category code 12 (“other”), including the backslash that \string inserts to represent an escape character. However, category 10 will be assigned to the character ‘ ’ (blank space) if a space character somehow sneaks into the name of a control sequence.

So I guess my question is: Is the claim from the TeX book quote above simplified? Then how exactly does the current value of \escapechar impact the expansion of \string?

Answer 1

The \escapechar setting means that \string\foo is foo not \foo (no backslash) this means that the \gobbletwo gobbles if not \i when applied to \string\ifzzz

The texbook quote is assuming the default value of \escapechar when it says that a \backslash_{12} will be inserted.

Answer 2

The command \string<token> works in two different ways:

• if <token> is a character token, the same character with category code 12 is delivered (but 10 is used if <token> is an explicit space token);

• otherwise <token> is a control sequence and in this case the name of the control sequence is delivered preceded by the character with code \escapechar; all characters so produced will have category code 12, except for spaces that retain their usual category code 10; however, if \escapechar has negative value or beyond 255 (0x1FFFFF for Unicode engines such as xetex or luatex), no character will be prepended.

Thus, when \escapechar is −1, the result of \string\foo will be foo (all characters having category code 12).

This is completely independent of the current character having category code 0. So, if you have

\catcode`@=0
@string@foo

and all other settings are standard, the delivered string of characters will be

\foo

because the default value for \escapechar is 92, as set by TeX at startup. This is witnessed by the following interactive session

> tex -ini '\catcode`{=1\catcode`}=2\message{\the\escapechar}'
This is TeX, Version 3.14159265 (TeX Live 2019) (INITEX)
92

However, nobody can know what's the value of \escapechar when \newif is called. Since one is expected to do \newif\iffoo, the macro needs to strip the possible \escapechar from the output of \string\iffoo in order to be able to define \footrue and \foofalse.

For instance, when reading .fd files, LaTeX enters a state where \escapechar is set to −1, so the backslash will not be inserted when using \string (it's not the only place). So the safest strategy is to temporarily set \escapechar to a known value and restore the value it had afterwards. Therefore, the current value is stored in \count@ (note that \escapechar is an internal integer register), to be restored at the end of the job.

Setting the (temporary) value to 92 would require to use \@gobblethree (which is not defined in the kernel), so it's simpler to set it to −1 and use \@gobbletwo.

This could be done in a different way without storing the value in a counter, which might even be more preferable because \count@ is not affected.

\def\newif#1{%
  \let#1\iffalse
  \@if#1\iftrue
  \@if#1\iffalse
}
\def\@if#1#2{%
  \begingroup\escapechar=\m@ne\expandafter\endgroup
  \expandafter\def
    \csname\expandafter\@gobbletwo\string#1\expandafter\@gobbletwo\string#2\endcsname
    {\let#1#2}%
}

How does it work? The value of \escapechar is changed inside a group and will be restored as soon as the group ends. However, \expandafter pokes after \endgroup before the group ends so the token from \csname is formed when \escapechar to −1; \endgroup will restore the previous value of \escapechar and the \def will be performed.

An important difference from the similar macro in plain TeX is that LaTeX performs no check whether the argument to \newif is a control sequence whose name begins with if.

Answer 3

The quote you reproduced is indeed simplified, but further on the same page (40), the TeXbook reads:

In the examples so far, \string has converted control sequences into lists of tokens that begin with \₁₂. But this backslash token isn't really hardwired into TeX; there's a parameter called \escapechar that specifies what character should be used when control sequences are output as text. The value of \escapechar is normally TeX's internal code for backslash, but it can be changed if another convention is desired.

Anyway, when you want to be sure, better also check later chapters such as the three Summary of ... Mode chapters at the end of the book. In this case, in chapter 20, Definitions (also called Macros):

\string〈token〉. TeX first reads the 〈token〉 without expansion. If a control sequence token appears, its \string expansion consists of the control sequence name (including \escapechar as an escape character, if the control sequence isn't simply an active character). Otherwise the 〈token〉 is a character token, and its character code is retained as the expanded result.

Also, according to page 308, the omission of the escape char in Knuth's TeX when using \string happens if, and only if \escapechar is outside the [0,255] range:

There is, however, another solution: If TeX's \escapechar parameter—which will be explained in one of the next dangerous bends—is negative or greater than 255, then \string\\ works.

(the objective is to output a single backslash character token with category code 12).