In this question I am not asking for code.

I'm asking before I start implementing: Which direction do you think makes the most sense?

I am about to define a fully expandable macro to be used with LaTeX 2ε which in some sort of tail-recursive loop examines its argument token-wise.

One requirement is: In case an expandable token is found, it shall be expanded during examination unless it is preceded by \noexpand or \protect and/or is defined in terms of \protected.

The case of the token being defined in terms of \protected can be handled by examining the \meaning. The tail-recursive loop can be implemented with an argument holding a "flag"-token indicating whether the token processed in the previous iteration either was the control-word-token \protect or had the same meaning as the \noexpand-primitive.

Checking for \noexpand-meaning and "setting the flag for the next iteration" can be handled by an \ifx-comparison. (You may probably "copy" the \noexpand-primitive at a moment in time where it is definitely not redefined yet.)

The question that remains is about handling edge cases:

  • Somebody might temporarily have redefined the control-word-token \protect so that temporarily it is not a component of LaTeX 2ε's protection-mechanism.
  • Somebody might very temporarily use a token \protectcopy (which was \let equal to \protect at some stage of processing).

How would you check whether the token \protect is currently used as a component of LaTeX 2ε's protection-mechanism?

How would you check whether some other token currently is used in the same way in which in LaTeX 2ε usually the token \protect is used?

I tend to do two things:

  1. Check for the control-word-token \protect itself, e.g., via delimited arguments.
  2. Check for one of the meanings which \protect can have when used as component of LaTeX 2ε's protection-mechanism.

Would this be sufficient?

Is the condition of the token in question both being the control-word-token \protect and having one of the meanings which \protect can have (\relax / \string / \noexpand / \noexpand\protect\noexpand ) sufficient for assuming that the token in question is used as a component of LaTeX 2ε's protection-mechanism?

Btw: What is expl3's equivalent to \protect ?

  • Are we allowed to do string-based tests? For \text_expand:n, which does things similar to your aim, we look for the token \protect itself rather than the meaning. (One sees the same with any 'marker' that might be common, for example tokens equal to \relax or \@firstofone.)
    – Joseph Wright
    Oct 6, 2020 at 12:23
  • Ah, I see you've mentioned that, though not quite the way I'd do it (using \pdfstrcmp). Honestly, I don't think you should worry about people 'rolling their own': only the token \protect itself gets handled by e.g. \protected@edef, so making a copy is pretty redundant.
    – Joseph Wright
    Oct 6, 2020 at 12:26
  • 1
    I realize you have accepted an answer, but I was wondering if you could comment a bit more on the actual desired task? My tokcycle package is made for going through input streams and handling one token at a time, where you write the code on how to "handle" characters, groups, macros, and spaces. Oct 6, 2020 at 14:52
  • 1
    The use of \expandedtokcycle would allow you to fully expand the argument before getting processed by tokcycle. Then, the \Characterdirective could be set to flag a + or - in the expanded input stream. p.s. I hope you recover fully and quickly. Oct 6, 2020 at 15:13
  • 1
    Here's a "sort of" example: \documentclass{article} \usepackage{tokcycle} \newcommand\checkpm[1]{\gdef\pmTF{F}\expandedtokcycle {\ifx+##1\gdef\pmTF{T}\else\ifx-##1\gdef\pmTF{T}\fi\fi} {\processtoks{##1}}{}{}{#1}} \begin{document} \resettokcycle \def\B{\noexpand\rule{1pt}{\A}} \def\C{Here is a rule: \B} \def\A{+3pt} \checkpm {Test the macro \C and other \textit{content} \today.} +- present? \pmTF \def\A{3pt} \checkpm {Test the macro \C and other \textit{content} \today.} +- present? \pmTF \end{document} Oct 6, 2020 at 16:08

1 Answer 1


In the code for \text_expand:n, which aims to take 'broadly text' and expand any hidden content, the approach taken is as follows.

First, note that there is an importance to the order one works. In particular, expansion of macros or primitives has to be considered after various 'filtering' tests.

The \protect token in LaTeX2e has to have exactly that name in order to work with \protected@edef and so on. As such, \text_purify:n tests for it by name not definition: this is done using \pdfstrcmp as this makes string-based expandable tests easy. Other tokens with equivalent meaning are thus not picked up by this process.

Rather than use a flag, if one finds a \protect, one can grab the next token and handle it in one loop pass. This ability to 'look ahead' is needed for any expansion anyway. In \text_purify:n, I choose to re-combine \protect\foo[space] into \foo if that construct exists - a little effort but 'undoes' an otherwise-confusing expansion.

One needs to expand not only macros but also expandable primitives in such a scheme. That deals with 'raw' \noexpand, which will do it's job such that the next token will be treated as \relax.

In expl3 terms, there is no direct equivalent of \protect as e-TeX is required. Thus non-expandable macros may be \protected, and the availability of more machine cycles means this type of token-ise analysis is feasible. This means that expl3 tokens should always be 'safe' inside an \edef or similar. (Internal macros as well as the documented API follows the same rule: either \protected or safe to use in an expansion context.)

  • I think you might want to read l3text.dtx for the comments on things like encoding-specific commands: hopefully the ideas are useful
    – Joseph Wright
    Oct 6, 2020 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.