Macro techniques using e-TeX

Question

The e-TeX extensions were intended to address some of the shortcomings of Knuth's original TeX, and since they are required by LaTeX2e for a while now, most package authors can rely on their presence. Now while there are some resources available for learning pure TeX macro idioms, including this discussion on this site, and also on the benefits of e-TeX in general, I would like to see some concrete examples of how e-TeX extensions can make the life of a macro writer easier.

In particular, some practical advice/tricks on the use of the expansion and parsing related commands \protected, \unexpanded, \detokenize and \scantokens would be much appreciated. They look useful to me, yet I never realize when I could use them to simplify things (the e-TeX manual is also very terse on them).

---

The scope of the question is fairly broad, but I would be most interested in the extensions listed above. Following are some further thoughts on other extensions that I would also be happy to discuss.

For some of the extensions, their usefulness is immediately evident to me, including

• not having to worry about allocating new registers due to the increased number of 32767 slots per type
• the convenient \numexpr, \dimexpr etc. expression commands facilitating arithmetic operations
• \unless as a negated \if, especially in conjunction with \loops where otherwise new conditionals would have to have been defined if a loop was to be executed as long as a conditions was false
• various new tracing commands and options, helping with diagnostics

But then, some features make me wonder whether they are actually used by packages out there or if they have come to be replaced by more "modern" approaches, such as

• mixed direction typesetting with \beginL, \beginR etc.
• status enquiries like \currentgrouptype, \currentiftype, \lastnodetype etc.
• the possibility of saving discarded items from the top of pages being built in \pagediscards
• line specific penalties with \interlinepenalties and the like

Answer 1

Taking the 'big ticket' items in e-TeX:

• \protected lets us create macros that do not expand inside an \edef or similar:

  \protected\def\foo{\let\baz\bong}
  \edef\test{Some text\foo}
  

  would normally give an error (assuming no definition for \baz/\bong), but here is 'safe'. If you look at the LaTeX2e \protect mechanism, the above is a lot easier to use and more reliable: \protected macros never expand inside x-type definition, whereas with the LaTeX2e approach one needs \protected@edef, etc.

• \unexpanded lets us protect arbitrary tokens from expansion without needing to use a toks

  \edef\test{Some text\unexpanded{lots of \textbf{stuff}}}
  

  which is very useful with arbitrary user input. It also allows things like

  \edef\demo{\unexpanded{tokens #}}
  

  which means one can easily store # inside a macro (see the expl3 tl datatype).

• \detokenize is good for making stuff 'safe', for example if it might have active chars

  \csname Tokens\detokenize{&_~^\foo}\endcsname
  

  which is particularly true for example if using active chars for UTF-8 input in LaTeX

• \numexpr, etc., work great for doing expandable calculations easily

Answer 2

grepping through the source files in TeX Live, I can add some insights on the use of the at least in my perception less widespread features of e-TeX.

For some of the status enquiry primitives, Heiko Oberdiek's packages provide some illustration for their application. The atbegshi package shows a neat trick how a primitive taking a box as an argument can be redefined (\shipout in this case), while taking into account that if the argument is to be intercepted with \setbox and \afterassignment, the token to be inserted after the assignment may or may not land inside the box depending on whether the box is given as box0 or as \hbox{...}. It looks more or less like this:

\def\shipout{%
  \edef\saved@grouplevel{\number\currentgrouplevel}
  \afterassignment\@test 
  \setbox\mybox=}

\def\@test{%
  \ifnum\saved@grouplevel<\currentgrouplevel
    \expandafter\aftergroup
  \fi
    \output
}

It uses e-TeX's \currentgrouplevel to determine whether the \@test landed inside the box (which opened a group), in which case the ensuing \output operation is postponed until after the box.

Another useful fact is that \lastnodetype is less than 11 for all non-discardable things, and -1 if the current list is still empty (see for example setouterhbox for applications of these).

As egreg points out in the comments, \currentgrouptype can be queried in math mode to decide whether to use a \middle delimiter variant or not, as \currentgrouptype is 16 in a group opened by \left (for example here).

e-TeX's bidirectional typesetting capabilities in the form of TeX--XeT is what is used by XeTeX, for which the bidi package provides an interface to these features. Also for some standard LaTeX packages like babel-hebrew this seems to be the way to go. However, LuaTeX has abandoned the \beginL, \beginR etc. primitives and integrates the Omega TeX extension instead.

Then again, for several features I haven't found any real applications. The only places where they show up are where probably all primitives will show up, like the LaTeX3 kernel, ConTeXt MkIV and the unravel package, and in a few other places where authors tried to increase compatibility with other packages using them. These features include the \currentif{type,branch,level} enquiries, the \pagediscards functionality and the line specific \{inter,club,widow}linepenalties.