# What tokens are in the end of line?

Consider the following test.tex:

\ERROR
a


If we run tex on it, and start skipping tokens one by one, we get this:

(test.tex
! Undefined control sequence.
l.1 \ERROR

? 1
l.2 a

? 1
l.2 a

? 1
)


As we see, the first 1 command skips end-of-line. Then we see that TeX has just read a and is about to read end-of-line character. We skip it with the second 1 command.

Then we see the same picture. As if nothing happened. And only after the third 1 command we see that TeX has passed further.

Why TeX skips end-of-line after \ERROR from the first time, but after a only from the second time?

Also, it is strange that if after the final *, which is not shown in the output, we say \end, tex says:

*\end
<*> \end

?


EDIT

Consider the following 6 test files:

Here are results that I obtained:

control-word_at_eol.tex: space and ^^M is skipped right after reading control word

control-word_not_at_eol.tex: space is not skipped right after reading control word

non-space_at_eol.tex: space is skipped right after reading control symbol, ^^M not skipped

non-space_not_at_eol.tex: space is not skipped right after reading control symbol

space_at_eol.tex: space, ^^M and (!!!) name of control symbol are skipped right after reading control symbol

space_not_at_eol.tex: space is not skipped right after reading control symbol

• correction) in file 2) space is skipped, it's ^^M which is not, in file 3) space not skipped it's ^^M. file 6) is same as 2 space skkiped but ^^M not. – touhami May 22 at 23:07

You can understand this better with a different input

\ERROR xyz
a


If you run tex on it, you get

This is TeX, Version 3.14159265 (TeX Live 2019) (preloaded format=tex)
(./igor.tex
! Undefined control sequence.
l.1 \ERROR
xyz
? 1
l.1 \ERROR x
yz


Error lines are divided into two parts (actually three if TeX is in the middle of expansions when it finds an error); here the top line shows how far TeX has gone in the input file, the bottom line shows what's awaiting to be read. Note that \ERROR sits in the top line, so it has already been read and gone. The slight misalignment between the two lines is because TeX represents control words with a trailing space, but the space “is not there” and, anyway, it is no longer relevant because it's in the top line.

With 1 you delete the next token and TeX stops again showing the context as before.

\ERROR
a


Let's see it step-by-step:

! Undefined control sequence.
l.1 \ERROR

?


Nothing is shown in the bottom line, because the line has ended. The end-of-line has already been converted to a category code ^^M and the generated space has been gobbled during tokenization of \ERROR. TeX is in state N (beginning of a new line).

If you type 1 followed by return, you get

? 1
l.2 a

?


TeX has entered state M because it has read a (and ignored it); the bottom line shows nothing for the same reason as before. Note that the ^^M is still unread.

Type another 1 to get

? 1
l.2 a

?


The gobbled token is now the space generated by ^^M and TeX enters state N again.

Type another 1:

? 1
)
*


The ) means the file has ended; * means that TeX is awaiting for input. This is again in the format “top line/bottom line”: there was nothing to be ignored, but TeX follows instructions and wants to ignore something.

Type \end:

<*> \end

?


Again, “top line/bottom line”: the top line shows what has been read in so far. There is no line number, hence TeX shows <*> to denote something directly input in the interactive session and what it has ignored following the previous 1 instruction.

Hitting return now shows the * prompt: TeX is awaiting for input.

• Top line does not end with a space in "xyz" example: try double click in terminal at the end of top line - highlighting will begin right after \ERROR, thus there is no space. – Igor Liferenko May 22 at 1:17
• Ideally, in the original example the ^^M converted to space should have been shown at the end of top line after second 1 command (after a), because the top line shows how far TeX has gone in the input file. The absence of space can be verified objectively by running hexdump -Cv test.log – Igor Liferenko May 22 at 1:47
• Please fix "The top line ends with a space" sentence, as it is wrong. See answer to exercise 8.3 in TeXbook. – Igor Liferenko May 22 at 2:58
• See EDIT in OP. – Igor Liferenko May 22 at 3:32
• @IgorLiferenko It was not wrong; I reworded anyway. About the six files, I don't know what to say: I'm not going to download them from an unknown repository – egreg May 22 at 8:43

Why TeX skips end-of-line after \ERROR from the first time, but after a only from the second time?

TeX removes any spaces from the end of the line (some older implementations erroneously removed tabs as well) and then if \endlinechar is in the 0-255 range that character is added to the end of the line) in plain and latex that defaults to 13 (control M).

As ^M has catcode 5, it normally acts as space so is gobbled while tokenizing \ERROR and which explains the different behaviour for the first two lines.

if you delete 2 tokens then \end works

! Undefined control sequence.
l.1 \ERROR

? 1
l.2 a

? 1
l.2 a

?
)
*\end
No pages of output.


using pdftex here.

with three 1 you get

! Undefined control sequence.
l.1 \ERROR

? 1
l.2 a

? 1
l.2 a

? 1
)
*\end
<*> \end


because the final 1 gobbles the next token but that is the \end that you are about to add....

• The difference is because you do not type 1 after the last ?. – Igor Liferenko May 21 at 9:38
• @IgorLiferenko yes I just updated the answer – David Carlisle May 21 at 9:39