# Does TeX always insert the macro \par?

When the input processor encounters two characters with category code 5 in a row (in other words, a blank line), it inserts the \par macro.

When \vbox{Abc.} ends, TeX ends the current paragraph, but not by inserting the \par macro. It seems that TeX is inserting the \par primitive instead. Am I understanding correctly what TeX is doing in this case? What about other places where TeX inserts \par? When is it the macro \par, and when the primitive?

\catcode@=11
\let\@@par\par
\def\par{\typeout{Macro!}\@@par}

Abc.

\vbox{Abc.\tracingall}

-
Best read on these things tug.org/TUGboat/tb14-3/tb40rycko.pdf – Yiannis Lazarides Dec 4 '11 at 15:47

There are exactly 7 places in the TeX program where TeX executes the paragraph builder internally, i.e. turning a horizontal list (if there is one under construction) into a paragraph. This happens not by inserting a \par token into the input stream but by executing the procedure end_graf implemented in module §1096 in the TeX program.

This procedure does nothing if TeX is not in horizontal mode, more or less nothing if in horizontal mode but with an empty list (null paragraphs are ignored by TeX), and executes the procedure line_break otherwise (and that one does all the magic about adding parfillskip penalties, etc.).

The seven places are

• at the end of the internal vertical structures like the closing brace of a \vbox but also \noalign or \vcenter or an alignment cell
• when the primitive par_end is sensed (which is available on the macro level initially as the meaning of the token \par)
• immediately after an output routine (OR) has ended (thus any horizontal list started in the OR will not continue with material from the galley but form a paragraph by its own)

In none of these cases there is a \par token inserted (that might be subject to redefinitions); instead, the end_graf procedure is executed!

\par tokens are only inserted when in horizontal mode and primitives incompatible with horizontal mode (like \vskip, \hrule, ... the full list in is §1094 in the TeX code) are encountered. And of course in the tokenization process when TeX replaces two endline chars by \par (i.e., making empty lines equivalent to \par).

-
Thanks for joining:) – Yiannis Lazarides Dec 17 '11 at 8:32
Hello Frank, as we say to new people 'Welcome to TeX.sx!' :-) You can use backticks  to mark your inline code as I did in my edit (I decided internal TeX procedures also counted: I hope this makes sense). – Joseph Wright Dec 17 '11 at 8:45
Thank you Frank, this is a very complete answer. Going back to the Web code is indeed the right thing to do for this kind of questions. – Bruno Le Floch Dec 17 '11 at 8:59
@Bruno I said 7 places where end_graf procedure is used. I was not counting the detailed usages in each place. For example in §1094 a number of actual usages are covered. Anyway, it is not the usage of par_end (which is the primitive \par) to look for but end_graf. On the other hand I used the index of the printed program (so perhaps that is not correct ;-) – Frank Mittelbach Dec 17 '11 at 10:23
it also covers the cases where TeX inserts a \par token in response to seeing certain commands; the other places where end_graf is used are §1026 §1085 §1100 §1131 §1133 §1168 – Frank Mittelbach Dec 17 '11 at 11:31

According to The TeXbook, the rule for end-of-lines is as follows

If TeX sees an end-of-line character (category 5), it throws away any other information that might remain on the current line. Then if TeX is in state N (new line), the end-of-line character is converted to the control sequence token \par (end of paragraph); if TeX is in state M (mid-line), the end-of-line character is converted to a token for character 32 ( ) of category 10 (space); and if TeX is in state S (skipping blanks), the end-of-line character is simply dropped.

Here, it's important to note that the TeX inserts the \par token, and not the \par primitive. That means that \par has to be defined under all circumstances. Knuth illustrates this point when discussing commands which force horizontal mode:

The appearance of a <vertical command> in restricted horizontal mode is forbidden, but in regular horizontal mode it causes TeX to insert the token \par into the input; after reading and expanding this \par token, TeX will see the <vertical command> token again. (The current meaning of the control sequence \par will be used; \par might no longer stand for TeX's \par primitive.)

Now, what does not seem to be mentioned anywhere is what happens at the end of internal vertical mode. Clearly, the rules above do not allow for the insertion of a \par token as there are no end-of-line markers to deal with. (You do get a \par inserted if there is a blank line between the last material in the \vbox and the end of the box.)

Reading the trace output, there is no mention of the \par primitive at the end of the box, but the shipout does show

....\penalty 10000
....\glue(\parfillskip) 0.0 plus 1.0fil


and of course there is clearly a paragraph built. So my conclusion is that the end of internal vertical mode implicitly inserts a \par primitive, and thus inserts the usual end-of-paragraph material and then runs the paragraph builder.

-
Very nice reply. It can be improved if you define what \par token, \par primitive, restricted horizontal mode, regular horizontal mode, internal vertical mode are. Of course, you can say, read the book. – Sony Dec 4 '11 at 14:10
@Sony Given the nature of the question (low-level TeX), I've taken the approach that an answer can make certain assumptions about the knowledge of the reader. This is of course a tricky balance! – Joseph Wright Dec 4 '11 at 14:32
@JosephWright Is there any way to access the \par token? – Yiannis Lazarides Dec 4 '11 at 14:59
@YiannisLazarides I'm not sure I follow. The \par token is just \par, and may in principal have any definition. The \par primitive is saved by LaTeX as \@@par, or is accessible using \pdfprimitive\par with a recent pdfTeX or LuaTeX, or \primitive\par with a recent XeTeX. – Joseph Wright Dec 4 '11 at 15:11
Plain TeX, LaTeX and ConTeXt also makes a copy of the \par primitive as \endgraf – Joseph Wright Dec 4 '11 at 15:21

There is a reason why TeX puts the "original meaning" of \par at the end of a \vbox: consider the following silly input

\vbox{\let\par\empty a}


When TeX finds }, it backs it up, terminates horizontal mode by inserting the "original \par" and then rereads the } that closes the internal vertical mode.

We can emulate what would happen if TeX inserted the current meaning of \par before rereading the } by

\vbox{\let\par\empty a\vskip0pt}


because \vskip inserts a (current) \par and TeX rereads \vskip. The result is an infinite loop!

(Found in a 1993 discussion on comp.text.tex.)

-
Same infinite loop happens before \end in horizontal mode, so I don't think this is much of a reason. I suspect the main reason is probably some implementation detail, and I agree with Frank Mittelbach that this may be a design mistake in TeX. – Bruno Le Floch Jan 18 '12 at 21:34
It may indeed be a design mistake: a "dead cycles" control should have been added for the automatically supplied \par tokens when a vertical command is sensed in horizontal mode. – egreg Jan 18 '12 at 21:42