Is there a complete modern book which explains how a TeX parser works?

Question

Hi I am reading 'The TEXbook' by Donald E. Knuth. But in my point of view is very didactic for people who want to learn TeX. Most of the pages contain exercises. But many things of how a TeX implementation parses a TeX code was untouched. I was wondering why the control sequence \input MS reads the argument 'MS' and not only 'M' as I suspected ($\sqrt ab$ is the square root of a and b is outside the root..) for instance.

In which order does a TeX parser process the characters of each of the 16 categories, which are listened on chapter 7....?

Does anyone know a complete reference or book which tells exactly how a TeX parser works and lists all the 900 build-in control sequences in TeX?

Answer 1

The explanation about \input is in the TeXbook, page 278:

The syntax for 〈file name〉 is not standard in TeX, because different operating systems have different conventions. You should ask your local system wizards for details on just how they have decided to implement file names. However, the following principles should hold universally: A 〈file name〉 should consist of 〈optional spaces〉 followed by explicit character tokens (after expansion). A sequence of six or fewer ordinary letters and/or digits followed by a space should be a file name that works in essentially the same way on all installations of TeX. Uppercase letters are not considered equivalent to their lowercase counterparts in file names; for example, if you refer to fonts cmr10 and CMR10, TeX will not notice any similarity between them, although it might input the same font metric file for both fonts.

In short, \input starts expanding tokens, ignoring spaces, until finding a nonspace unexpandable token which should be a character token (or a control sequence \let to a character token). Everything up to the first noncharacter token will be considered as part of the file name (expansion continues to be performed).

The .tex extension will usually be appended if not found in the specified file name (but this is implementation dependent; for example, Textures allowed file names without extension).

So bizarre code such as

\let\aletter=a
\input }{\aletter x

will try to input a file called }{ax.tex. A space token after the file name will be ignored.

Particular implementations can provide slightly different conventions; for instance, TeX Live allows the file name to be quoted between "; a leading " will make TeX look for another one (always doing expansion) and the file name will consist of every character token (including space tokens) until the trailing "; the two double quotes characters will be stripped off.

With XeTeX and LuaTeX still different conventions apply. For instance, LuaTeX allows \input {file} and will input file.tex; to the contrary, pdftex will try inputting {file}.tex.

Note that the same rules apply whenever TeX is looking for a file name, that is, for \openin, \openout and \font. However, XeTeX uses special conventions for the file name in \font that aren't applicable to other cases in which a 〈file name〉 is being looked for.

There are several other cases in which arguments to primitives are not braced: for instance

\dimen 123 = 1234pt

will not take just 1 as the register, but will go on (with macro expansion) until finding something that's not a digit (in this case the space).

In essentially all of these situations, a trailing space token will be swallowed, being considered a delimiter of the required tokens.

Answer 2

There is Knuth’s

Computers and Typesetting, Volume B, TeX: The Program

which is a “documented listing of the source code of the TeX interpreter.” (that is, the WEB (or Pascal) program)

But firstly you already have the right book at your hands: The TeXbook. As Joseph said, it indeed answeres your question about \input.