In chapter 8 "The Characters You Type", the TeXbook (20th printing, Addison-Wesley 1991) describes TeX's tokenizer, whose input is a stream of characters, and whose output is a stream of tokens. The tokenizer is described as a state-machine with three states:
- N - Beginning a new line
- M - Middle of a line
- S - Skipping blanks
Near the end of this chapter (on p. 47), it is described how the primitive
\input is dealt with by the tokenizer:
If TEX has nothing more to read on the current line, it goes to the next line and enters state
N. However, if
\endinputhas been specified for a file being
\input, or if an
\inputfile has ended, TEX returns to whatever it was reading when the
\inputcommand was originally given.
It is unclear to me:
what state change, if any, happens when TeX starts reading an
what state change, if any, happens when TeX resumes reading the main file after finishing reading the
whether the end of the
\inputfile is considered the end of the last line of that file.
Johannes_B and Joseph Wright gave good answers to questions 1 and 3, but the answer to question 2 is still not clear to me. Both Johannes_B and Joseph Wright claim that when the input file has been read, the tokenizer reverts to its former state. But at what point was this former state established? Before or after the
\input primitive was read? Before or after the file name was read? Before or after the spaces after the file name were consumed?