What combinations of mode and \currentgrouptype exist?

Question

I am trying to predict in any given situation what TeX does if a closing brace reaches its stomach (apart from reverting local assignments). For instance, \vbox{\tracingall H} shows that if it is in horizontal mode when closing a vbox, then TeX first cuts the current paragraph into lines before leaving the vbox group. One way to determine what TeX does is to list all possible combinations of group type and mode which can occur, and the corresponding action to be taken.

Is that a reasonable approach? If so, what is that list? Any better method? For reference, I put below the list of modes and group types.

TeX knows of six modes (see Chapter 13 of the TeXbook):

• Vertical mode.
• Internal vertical mode.
• Horizontal mode.
• Restricted horizontal mode.
• Display math mode.
• Math mode.

According the the eTeX manual, TeX knows 17 group types, and we can access that information through \currentgrouptype:

• bottom level (no group) (0)
• simple group (1)
• hbox group (2)
• adjusted hbox group (3)
• vbox group (4)
• vtop group (5)
• align group (6)
• no align group (7)
• output group (8)
• math group (9)
• disc group (10)
• insert group (11)
• vcenter group (12)
• math choice group (13)
• semi simple group (14)
• math shift group (15)
• math left group (16)

Answer 1

Let me answer the question of what happens when } is seen, one case at a time. TeX is in no mode within \write commands, which can occur in any kind of group, so I'll ignore this case.

The bottom level (where \currentgrouptype is 0) can occur in (outer) vertical mode or (outer) horizontal mode. It cannot occur in internal vertical mode nor in restricted horizontal mode, since boxes start groups. It cannot occur in either math modes since those start a group too. A closing brace in this case triggers the Too many }'s error.

1. simple group. Any mode. A closing brace ends the group.

2. hbox group. Restricted horizontal mode. A closing brace ends the group and the box.

3. adjusted hbox group. See hbox group, but the post-processing is different.

4. vbox group. Internal vertical mode. A closing brace ends any paragraph using the primitive \par, then ends the group and the box.

5. vtop group. See vbox group, but the post-processing is different.

6. align group (\halign{#\cr...} or \valign). Inside \halign, restricted horizontal mode; inside \valign, internal vertical mode or (outer) horizontal mode. A closing brace raises the Missing \cr inserted error, inserts a \cr, then the closing brace. This ends the alignemnt.

7. no align group (\noalign{...}). Inside \halign, internal vertical mode or (outer) horizontal mode; inside \valign, restricted horizontal mode. A closing brace ends any paragraph, ends the current horizontal or vertical mode, the group, and looks ahead for another \noalign, or \omit, etc.

8. output group. Internal vertical mode or (outer) horizontal mode. A closing brace often leads to an error, unless it is the closing brace inserted by \output itself, or it is skipped over with \expandafter. If the closing brace is ok, the group ends, the vertical material is put in the main vertical list, and TeX checks that \box255 is empty, raising an error otherwise.

9. math group (${...}$). (Inner) math mode. A closing brace ends the group, and builds an atom from the current math list.

10. disc group (\discretionary{...}{...}{...}). Restricted horizontal mode. A closing brace ends the current discretionary list (roughly the same as the contents of an \hbox, but with more restrictions); if this is the third argument of \discretionary, the three lists are measured, and a discretionary node inserted in the current horizontal list.

11. insert group (\insert0{...} or \vadjust{...} (thanks egreg)). Internal vertical mode, or (outer) horizontal mode (after starting a paragraph). A closing brace ends any paragraph by calling the primitive \par. Then \insert (or \vadjust) does its thing.

12. vcenter group (\vcenter{...}). See insert group, but the post-processing is different.

13. math choice group (\mathchoice{...}{...}{...}{...}). (Inner) math mode. A closing brace ends the current list, see discretionary group.

14. semi simple group (\begingroup...\endgroup). Any mode. A closing brace causes an error and is ignored.

15. math shift group ($...$ or $$...$$). Either math modes. A closing brace causes an error and is ignored.

16. math left group (\left(...\right)). Either math modes. A closing brace causes an error and is ignored.

Note that when vbox-like groups end, the primitive meaning of \par is inserted to end any paragraph, not the current meaning of the \par token. When a box ends, it is either put in the current list (\hbox{...}), or assigned to a box register (\setbox0\hbox{...}), etc. In all cases, if a group ends, \aftergroup tokens are inserted after the closing brace.

Answer 2

From the TeXbook, p. 279, in chapter 24, Summary of vertical mode

• }. This one is harder, because it depends on the current group. TeX should now be working on a group that began with {; and it knows why it started that group. So it does the appropriate finishing actions, undoes the effects of non-global assignments, and leaves the group. At this point TeX might leave its current mode and return to a mode that was previously in effect.

The group can be of any type from 1 to 13. Type 14 is started by \begingroup, type 15 by $ and type 16 by \left or \middle.

With types 1, 2, 3, 6, 8, 9, 10, 13 the mode doesn't change before TeX exits the group (it can change after leaving it). With types 4, 5, 7, 11, 12 TeX might change from horizontal mode back to internal vertical mode in order to complete the task at hand (never from internal vertical mode to horizontal mode).