Taking the example (following the edit), I'll start giving a very detailed view of the first couple of lines then move on a bit more rapidly. Dan has noted the fact that there is a bit of processing to normalise 'lines': I'm going to assume we can move on to the char-by-char reading of each line.
The bottom line here is that TeX reads the input from the beginning, working one char at a time to do tokenization. (Line-by-line stuff is as already noted needed to normalise line ends but isn't normally a concern at the 'TeX end'.) Each token is then processed further as appropriate.
Unit A
The first line starts with a %
, which TeX reads and tokenizes as a comment char. That means that it skips the rest of the line.
The second line starts with \
, which is the escape char. TeX therefore keeps reading to find a control sequence: either a single non-letter or one or more 'letter' characters. That's actually done one at a time, but again let's skip to the result: a token called \def
. (This is followed by a \
, which is left alone for the moment: at this stage what's important is that it's not a letter so terminates the formation of a control word.) TeX now looks up a definition for \def
to see what to do: it might be expandable, it might be executable, it might be an implicit char (with say \let\foo=a
), or of course it might be undefined. In this case, it's a primitive that is executed. The \def
primitive needs to be followed by a token to be defined, possibly an parameter text and then a replacement text. TeX therefore needs to find these, which means more tokenization.
The first thing after \def
is \
as already noted. This starts a control word, so TeX once again builds one up a character at a time: it finds \aa
terminated by [
. So \def
will define a macro called \aa
. TeX now tokenizes [
and as isn't a start-group token, is becomes the first char of a parameter text. TeX works along, tokenizing #
, 1
and ]
similarly before it comes to {
. This is a start-group token, so the parameter text is complete and is [#1]
. Thus \aa
will be defined such that it must be followed by a [
(catcode 12), then a variable part (#1
) then a ]
(catcode 12). TeX is now constructing the replacement text for \aa
. It's seen the {
and now starts tokenizing the content. There is a brace-matching requirement here, so we have to remember that each token TeX reads might be }
or equivalent, but that no expansion is going on. Thus \aa
ends up with replacement text a's value: #1
where #1
is a place-holder for whatever comes between the [
and ]
required when \aa
is used.
After the }
is a space (end-of-line converted to a space by TeX). As we are in vertical mode, the space does nothing and so we can move on. The remaining definition lines can be analysed in the same way: each is read a char at a time with tokenization, execution, etc., but that is hopefully clear and pretty tedious to write out!
After the line defining \dd
there is a blank line, which TeX converts to a token called \par
which is inserted and executed (I'm assuming there has been no code before the first line here: \par
could also be a macro to be expanded). We are in vertical mode so the \par
primitive (which I'm assuming) doesn't do anything significant.
Unit B
Unit B starts again with a comment line: I'll skip that one and future comment lines. TeX then finds a \
again and after a bit of work has the token \expandafter
which it looks up. This is an expandable token which causes TeX to skip over the next token and to (try to) expand the following one. To do that, TeX obviously has to find the next two tokens, which is does: both turn out to be \expandafter
. At this stage life gets a little more fun. As the new token to be dealt with is also expandable, the process continues. You have a chain:
\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter
\expandafter
\aa\dd
which is you read it carefully results in TeX getting from that very first \expandafter
to \dd
. Now, \dd
is a macro with no parameters but with a replacement text, and so TeX does exactly one expansion, replacing \dd
by \cc
and effectively giving us
\expandafter\expandafter\expandafter\aa\cc
(follow the expansion carefully to see this!).
(Note: in the above I've ignored the line endings. TeX has converted line ends to spaces, s mentioned earlier, and after control words TeX skips spaces. Thus they don't 'count' for working out what the next token is.)
TeX has already tokenized the above as part of the first \expandafter
chain, so there is no question about tokens here. We have another \expandafter
, so the same process is repeated, this time finding \cc
which again is a macro so is replaced, giving us effectively
\expandafter\aa\bb
There is still an \expandafter
, so yet again a replacement takes place to give
\aa[Hi]
TeX now has to expand \aa
, but this time there is a parameter text to allow for. Thus TeX matches up the input with the requirements: there is a [
with catcode 12, there is 'some stuff' (Hi!
) then there is a ]
with catcode 12. TeX now inserts the replacement text for \aa
with #1
replaced by Hi!
, so we have
a's value: Hi!
The letter a
is just a letter: reading it forces TeX to switch to horizontal mode. TeX then inserts the \everypar
token parameter, which is empty here, then starts collecting material to build a paragraph. As everything is now 'text', nothing 'new' happens until we get to the empty line. This inserts a \par
token, which again is the primitive. As we are in horizontal mode this will start the paragraph-building algorithm. That's a topic on it's own: for the moment, we can I hope take it that TeX builds a paragraph inserting various skips, checks that the paragraph will fit on the current page without breaking, and as it does adds it to the 'current page' material. It doesn't add it to the PDF/DVI yet.
Units C and D
These are very similar to unit A. Exactly the same analysis applies except there are fewer rounds of expansion. As I've noted above, each time \expandafter
is expanded we can view the input as 'rewritten' and 'simplified' as that is basically what does happen. As such, other than being less complex there is nothing new to say here.
Last line
The last instruction here is bye
. This is a tidy-up operation which finishes off the run. In particular, it will ensure that any material for the 'current page' is actually shipped out before finalising the DVI/PDF file. Thus with the short demo here this is the only place where any \shipout
will occur. The plain TeX output routine is quite simple, but it does add for example a page number to the current page. Thus \bye
is again probably best analysed on its own.
;-)