What state changes are caused by \input?

Question

In chapter 8 "The Characters You Type", the TeXbook (20th printing, Addison-Wesley 1991) describes TeX's tokenizer, whose input is a stream of characters, and whose output is a stream of tokens. The tokenizer is described as a state-machine with three states:

• N - Beginning a new line
• M - Middle of a line
• S - Skipping blanks

Near the end of this chapter (on p. 47), it is described how the primitive \input is dealt with by the tokenizer:

If TEX has nothing more to read on the current line, it goes to the next line and enters state N. However, if \endinput has been specified for a file being \input, or if an \input file has ended, TEX returns to whatever it was reading when the \input command was originally given.

It is unclear to me:

1. what state change, if any, happens when TeX starts reading an \input file.

2. what state change, if any, happens when TeX resumes reading the main file after finishing reading the \input file.

3. whether the end of the \input file is considered the end of the last line of that file.

---

Johannes_B and Joseph Wright gave good answers to questions 1 and 3, but the answer to question 2 is still not clear to me. Both Johannes_B and Joseph Wright claim that when the input file has been read, the tokenizer reverts to its former state. But at what point was this former state established? Before or after the \input primitive was read? Before or after the file name was read? Before or after the spaces after the file name were consumed?

Answer 1

When the \input primitive is encountered, TeX starts reading the new file and starts a new line for the beginning: thus the state changes from 'whatever it was' to N. It then proceeds under the normal rules until the end of the file (or \endinput). TeX will then return to whatever state it was in before.

In most cases, this is all somewhat academic as \input is typically used on a line on it's own with 'well-behaved' source files. Thus the outcome is the same as if the lines of the file input were simply copy-pasted into the main source. If we deliberately arrange that this is not the case we can do tests to show that the behaviour is as described. For example, with file main.tex reading

\def\world{world}
\def\bar{bar}
Hello \world \input foo \bar
\bye

and foo.tex reading

foo

we get output

Hello worldfoo bar

On the other hand, if we change foo.tex to read

foo%

then we get output

Hello worldfoobar

This shows that as described TeX has returned to skipping spaces after the end of the \input primitive. In the first case, there is a newline at the end of foo.tex which turns into a space, and so shows up. In the second case we don't have that space, and the one after \input foo is ignored so we don't see it. Notice that that space does delimit the file name picked up by \input: without it we need a `\relax.)

If the input file contains \endinput then all lines after that one are ignored, but not anything on the same line but after \endinput. Again, altering our demo foo.tex to

foo \endinput baz%
oops

we get

Hello worldfoo bazbar

or if we omit the %

Hello worldfoo baz bar

Answer 2

The following is meant to complement Joseph Wright's answer by offering a clear answer to question no. 2 in the original post, namely what state change, if any, happens when TeX resumes reading the main file after finishing reading the \input file. This answer is speculative, in that I don't base it on the source code, the TeXbook, or any other authoritative source, but on my own thoughts and observations. It is essentially what Joseph Wright wrote in his answer, except I don't claim that once the input file has been read, TeX will "return to whatever state it was in before". Other than that, it's basically a restatement of his answer.

I will assume a grammatically correct TeX manuscript. I will also assume that the input filename is specified explicitly rather than via a macro, and that it is ended with whitespace or some non-expandable control-sequence.

When the TeX "stomach" receives the token \input from the tokenizer, the characters that gave birth to this token have been consumed, and so has the following whitespace, so that the next character on the input stream marks the file name's first character.

Assuming the token \input has the default value (i.e. assuming it has not been \let to some other value), the next thing to happen is that the file name will be consumed, until the next character on the input stream will be whitespace, or some non-expandable control-sequence.

The tokenizer will then switch to state N (as pointed out in Joseph Wright's answer), the current input stream will be cached, and then the current input stream pointer will be directed to the input stream engendered by the input file.

When the latter input stream has been consumed, the tokenizer's state will change to S, and the current input stream pointer will be redirected to the cached input stream. Thus, any whitespace following the original input file name will now be consumed (as pointed out in Joseph Wright's answer).