Understanding Implicit Delimiters/Terminators

[I encourage you to check out Phelype's impressive approach]

I hear about macros that do things like "expand until they reach an unexpandable token" and this question may (or may not) be related to that kind of thing and the \romannumeral trick. In particular, I'm thinking of the behavior of things like \numexpr which terminate upon encountering a \relax token. However, they also terminate without a \relax token, if an end-of-group is reached (and apparently also when a begin-group is reached). Thus

\the\numexpr 1+1+1\relax


and

{\the\numexpr 1+1+1}


both work.

I have an interest in a recursive version of \numexpr, call it \rnumexpr, that will expand groups in its argument, continuing the calculation using the previously grouped data.

Here it is and it seems to work great. It relies on a feature of tokenization that if a group is passed as an argument, the grouping is stripped and the contents of the group become the actual argument.

However, with my coding, it requires an explicit terminator (in this case, \rrelax).

EDITED to handle up to 8 nesting levels (i.e., 8 successive left braces), but it still can't handle an implicit delimiter

\documentclass{article}
\makeatletter
\let\@relax\relax

% CAN HANDLE 8 SUCCESSIVE LEFT BRACES
\def\rnumexpr#1\rrelax{\numexpr\@rnumexpr
\@empty\@empty\@empty\@empty\@empty\@empty\@empty\@empty\@empty
#1\relax \@empty\@empty\@empty\@empty\@empty\@empty\@relax}

\def\@rnumexpr#1#2#3#4#5#6#7#8#9\@relax{%
#1\ifx\relax#2\relax\else\@rnumexpr#2#3#4#5#6#7#8#9\@relax\fi}
\makeatother

\begin{document}
\the\numexpr+1+1+1+1+1\relax,
\the\numexpr+1+1{+1+1+1}\relax,
\the\numexpr+1+1{+1{+1+1}}\relax

\the\rnumexpr+1+1+1+1+1\rrelax,
\the\rnumexpr+1+1{+1+1+1}\rrelax,
\the\rnumexpr+1+1{+1{+1+1}}\rrelax,
Expandable! \edef\z{\the\rnumexpr+1+1{+1{+1+1}}\rrelax}\z

\the\rnumexpr+1+1+1+1+1\rrelax,
\the\rnumexpr+1+1{+1+1+1}\rrelax,
\the\rnumexpr+1+1{+1{+1+1}}\rrelax,
\the\rnumexpr{+1{+1{+1{+1{+1{+1{+1{+1{+1{+1}}}}}}}}}}+1\rrelax,

Can handle up to 8 successive left braces:
\the\rnumexpr{+1{{{{{{{{+1}+1}+1}+1}+1}+1}+1}+1}+1}+1\rrelax{},
\the\rnumexpr{+1{{{{{{{{+1}}}}}}}}}+1\rrelax{},
\the\rnumexpr{{{{{{{{+1}}}}}}}}\rrelax{}

{\the\numexpr1+1+1} numexpr uses implicit delimiter

%{\the\rnumexpr1+1+1}
but rnumexpr won't work...EXPLICIT DELIMITER EXPECTED

\end{document}


The first two lines compare the results of \numexpr and \rnumexpr, showing how \numexpr appears to stop when it reaches the begin-group, whereas \rnumexpr extracts it and continues the calculation. It is even shown to be expandable!

The 3rd and 4th lines show put \rnumexpr to a tougher test. Phelype pointed out that my original request was quite limited as to how many levels of nesting it could handle. This edited approach can handle more nesting levels (up to 8 successive left braces), but still has a finite limit.

The 5th line of output shows how \numexpr can terminate without an explicit \relax. Attempting such a syntax with \rnumexpr does not work because I've coded it to expect an explicit delimiter.

Is there a way to redefine \rnumexpr to also end when reaching an end-of-group rather than an explicit terminator (while at the same time not ending when reaching a start-of-group)

Note: The purpose here is not to develop a logical approach to nested calculations. While that may be a desirable thing in certain applications, that is not what is being attempted here. Thus, approaches that suggest using parens rather than braced subunits do not address my concern.

As I replied to David, the process I am really interested in is counting certain "qualified" tokens across an arbitrary argument. Using the approach I am taking to this larger question, for example, I ignore "unqualified" tokens, but when I come across "qualified" tokens, I place a +1 in the output macro. However, the process I have developed also retains the grouping of the original argument in the output macro.

So when I am done examining the argument token-by-token (with grouping retained), the output contains an arbitrary number of +1 tokens within the argument's original grouping structure. It is this output macro that I hope to operate on with \rnumexpr. Since I am writing the code, I can always be sure that I add the \rrelax at the end, but this question has more to do with me wondering if it was possible to rewrite \rnumexpr without the closing delimiter.

• why do you want {} groups rather than () ? especially \rnumexpr+1+1{+1+1+1}\rrelax which looks like it should be an error rather than \rnumexpr+1+1+{1+1+1}\rrelax ? May 24 '19 at 15:09
• @StevenB.Segletes That can be easily done if the macro has a mandatory delimiter (be it \relax or brace-delimited, or anything else). With an optional delimiter like for \numexpr it could perhaps be done un-expandably (using \futurelet and a huge processing time). Both I think it's not possible without making the code extremely (and I mean it) fragile. Would you be okay with a mandatory delimiter? May 24 '19 at 15:31
• the note still doesn't clarify things that much, if you are just interested in a list of +1 possibly having ignorable brace groups that could be done easily enough, the general case with an arbitrary \numexpr expression with ignorable relax isn't feasible with anything like a reasonable amount of code. May 24 '19 at 15:59
• you would need something like Bruno's unravel package that re-implements the tex expansion model in tex macros, but then modify it to have a different expansion here ignoring groups, That is way way too hard to contemplate for an answer here, sorry. May 24 '19 at 16:08
• I don't think you could even do it with a reimplementation of the expansion model in this specific case. You would have to parse ahead until a token occurs that may not be used inside \numexpr, but I can't think of an expandable way to test whether that token is } without running into extra } errors. May 24 '19 at 17:48

I made you an expandable version of \rnumexpr which does not require a delimiter and will stop on the first unexpandable, \numexpr-invalid token. It tries to emulate the behaviour of \numexpr up to some extent, and ignore brace pairs.

The thing about \numexpr, which everyone already commented, is that it's a primitive, so its rules are different than the rules that govern the realms of men dealing with simple macros. Unfortunately some things simply can't be done without primitive support.

You want expandability, so right off the bat you cannot have lookahead (with \futurelet). \futurelet would allow you to look at the next token and decide what to do with it. Expandability restricts you to grab tokens as arguments and pass them around in funny ways, and grabbing stuff as argument (with a open-ended command like \rnumexpr) means that:

1. {\rnumexpr 1+1} is impossible because TeX will yell at you when it grabs }
2. \rnumexpr 1+1 ⟨something else⟩ will eventually grab ⟨something else⟩, whatever it happens to be, determine if it has to be expanded or not, and deal with it accordingly.

With a delimited argument you could use something like expl3's \__tl_act:NNNnn to expandably loop through a token list and act on an item differently, depending if it is a space, a grouped token list, or another single token, which would make the task at hand much easier.

First let me point some things about your code. In your test for emptiness \expandafter\ifx\relax#2\relax, the \expandafter skips \ifx and expands \relax, so it isn't of much use and can be removed. Also this test might print undesired characters should the the input contain a \relax. Of course you are in the middle of a \numexpr, so this is just nitpicking.

Also your conditional doesn't end at each iteration of \@rnumexpr, but only at the very end of the \numexpr. This will, for large expressions (and with large I mean enough copies of +1 to get a result larger than 1500–very large) use up all of TeX's input stack. And finally, your definition does not work for \rnumexpr{+1{+1}}+1\rrelax and other (far too weird to be considered normal input) combinations of braces.

I defined a slow, certainly-suboptimal, probably-too-convoluted, most-likely-buggy, ⟨insert-other-qualifiers-here⟩, emulation of \numexpr. Mostly the behaviour is the same (to the extent of the tests I done), except that it ignores braces.

It starts scanning the input, token by token, then deciding what to do with each. It tries to expand tokens as it goes, and stops on the first unexpandable, \numexpr-invalid token. If that token is \relax, it is consumed, like \numexpr does, so the behaviour is very similar in this aspect.

The major difference is that, as it grabs tokens as undelimited arguments, spaces are ignored, so while the result of \the\numexpr 1+1 1 is 21 (2 appended with an 1), the result of \the\rnumexpr 1+1 1 is 12 (1+11), so it needs a “harder” ending token than \numexpr. This can be avoided by either using a \relax: \the\rnumexpr 1+1\relax 1 to end the \rnumexpr or by using \obeyspaces so that the spaces are sent to the underlying \numexpr which will then do the right thing.

here it is:

\documentclass{article}

\makeatletter
\def\rnumexpr{\romannumeral-0\rn@collect{}}
\long\def\rn@collect#1#2{%
\rn@ifsinglechar{#2}%
{%
\rn@ifvalid@numexpr@token{#2}%
{\rn@collect{#1#2}}%
{\rn@finish{#1}{#2}}%
}%
{%
\rn@ifsingletoken{#2}%
{%
\rn@ifrelax{#2}%
{\rn@finish{#1}{}}%
{\rn@expand@after{#1}#2}%
}%
{\rn@collect{#1}#2}%
}%
}
\def\rn@qrtail{\rn@qrtail}
\def\rn@expand@after#1{%
\rn@@expand@after{\expandafter\rnumexpr}#1\rn@qrtail\rn@qrstop}
\def\rn@@expand@after#1#2{%
\ifx#2\rn@qrtail
\rn@finish@expandafter{#1}%
\else
\expandafter\rn@@expand@after
\fi
{#1\expandafter#2}%
}
\def\rn@finish@expandafter#1#2\fi#3\rn@qrstop{%
\fi#1\romannumeral-0\rn@check@unexpandable}
\long\def\rn@check@unexpandable#1{%
\expandafter\rn@@check@unexpandable\expandafter#1%
\romannumeral-0#1}
\long\def\rn@@check@unexpandable#1#2{%
\ifx#1#2%
\expandafter\rn@unexpandable
\else
\expandafter\rn@expandable
\fi
{#1}{#2}}
\long\def\rn@expandable#1#2{#2}
\long\def\rn@unexpandable#1#2{\relax#2}
\long\def\rn@finish#1#2{%
\numexpr#1\relax#2}
\long\def\rn@ifrelax#1{%
\ifx#1\relax
\expandafter\@firstoftwo
\else
\expandafter\@secondoftwo
\fi
}
\def\rn@ifvalid@numexpr@token#1{%
\expandafter\rn@@ifvalid@numexpr@token\expandafter{\number#1}}
\def\rn@@ifvalid@numexpr@token#1{%
\if
\ifnum58>#1    1\else x\fi
\ifnum   #1>39 1\else y\fi
\ifnum
\ifnum#1=44 1\else 0\fi
\ifnum#1=46 1\else 0\fi
=0
\rn@true
\else
\rn@false
\fi
\else
\ifnum#1=32
\rn@true
\else
\rn@false
\fi
\fi
}
\def\rn@true{\expandafter\@firstoftwo\romannumeral-0}
\def\rn@false{\expandafter\@secondoftwo\romannumeral-0}
\edef\rn@catofamp{\the\catcode\&}
\catcode\&=11
\long\def\rn@gobble#1&{%
\romannumeral-0\rn@@gobble#1\rn@qrtail &}
\long\def\rn@@gobble#1#2&{%
\ifx\rn@qrtail#1%
\expandafter\rn@@gobble@end
\else
\expandafter\rn@de@tail
\fi#2}
\def\rn@@gobble@end{ }
\long\def\rn@de@tail#1\rn@qrtail{ #1}
\long\def\rn@ifsinglechar#1{%
\rn@ifempty{#1}%
{\@secondoftwo}%
{%
\if\relax\expandafter\rn@gobble\detokenize{#1}&\relax
\expandafter\@firstoftwo
\else
\expandafter\@secondoftwo
\fi
}%
}
\long\def\rn@ifsingletoken#1{%
\rn@ifempty{#1}%
{\@secondoftwo}%
{%
{\@secondoftwo}%
{%
\if\relax\detokenize\expandafter\expandafter
\expandafter{\rn@gobble#1&}\relax
\expandafter\@firstoftwo
\else
\expandafter\@secondoftwo
\fi
}%
}%
}
\ifcat\expandafter\@gobble\expandafter{\expandafter{\string#1?}}**%
\expandafter\@secondoftwo
\else
\expandafter\@firstoftwo
\fi
}

\catcode\&=\rn@catofamp
\long\def\rn@ifempty#1{%
\if\relax\detokenize{#1}\relax
\expandafter\@firstoftwo
\else
\expandafter\@secondoftwo
\fi
}
\makeatother

\begin{document}

\def\twop{+1+1}

\the\numexpr 1+1 1

\the\rnumexpr 1+1 1

\the\numexpr\twop+1+1+1
\the\numexpr\twop+1+1+1
\the\numexpr\twop+1+1+1
\the\numexpr\twop+1+1+1+1+1
\the\numexpr\twop+1+1+1+1+1

\the\numexpr 1+1
\the\numexpr 1+1\twop

\def\twop{{+1+1}}

\the\rnumexpr\twop+1{+1+1}\relax
\the\rnumexpr\twop{+1+1+1}\relax
\the\rnumexpr\twop{+1{+1+1}}\relax
\the\rnumexpr\twop{+1{+1+1}}+1+1\relax
\the\rnumexpr\twop{+1{+1+1{}}}+1+1\relax

\the\rnumexpr 1+1
\the\rnumexpr 1+1\twop

Expandable! \edef\z{\the\rnumexpr+1+1{+1+1}\relax}\texttt{\meaning\z}

\the\rnumexpr1{{+1}+1{+1}}+1\relax

\the\rnumexpr{1{+1}}+1\relax

{\the\numexpr1+1+1}

Groups everywhere:
\the\rnumexpr{+1{+1{+1{+1{+1{+1{+1{+1{+1{+1}}}}}}}}}}+1,
\the\rnumexpr{+1{{{{{{{{+1}+1}+1}+1}+1}+1}+1}+1}+1}+1,
\the\rnumexpr{+1{{{{{{{{+1}}}}}}}}}+1,
\the\rnumexpr{{{{{{{{{{{{{{{{{{{{{{{{{{+1}}}}}}}}}}}}}}}}}}}}}}}}}}

No leftover:
\detokenize\expandafter{\the\rnumexpr{+1{{{{{{{{+1}}}}}}}}}+1\relax}

% {\the\rnumexpr1+1+1} STILL WON'T WORK :(

\end{document}


The macro could be much faster if the expression were evaluated with \the\numexpr0 beforehand, instead of grabbing every single token and evaluating them only at the bitter end. However this would spoil “stability” (if you can call it that) of the macro because at each evaluation (as many as there are groups), a \relax would be consumed, so to properly terminate the macro you would need to resort to things like \the\rnumexpr1{+1{+1{+1}}}\relax\relax\relax\relax, so I opted out of this possibility.

• Amazing. If I thought I was opening such a can of worms, I would have thought better of it. Give me some more time to examine your solution, but I am certain you will receive the check mark when the time comes. Thank you for the lesson! May 27 '19 at 16:41
• @StevenB.Segletes Usually these cans are the most fun to open! Glad it helped :) May 27 '19 at 17:50
• Thanks again for this effort. You surely deserve more than two upvotes! May 29 '19 at 10:25
• @StevenB.Segletes Oh, wow, nice! It tied a knot in my brain :-) I didn't exactly understand how it works, but if you change to \def\rnumexpr#1\rrelax{\numexpr\@rnumexpr \@empty\@empty\@empty\@empty\@empty\@empty\@empty\@empty\@empty #1\relax \@empty\@empty\@empty\@empty\@empty\@empty\@empty\@empty\@empty\xss@relax } \def\@rnumexpr#1#2#3#4#5#6#7#8#9\xss@relax{% #1\ifx\relax#2\relax\else\@rnumexpr#2#3#4#5#6#7#8#9\xss@relax\fi} (replacing the dummy \xss@relax by \@empty and using \xss@relax only as delimiter), you avoid leaving a bunch of \relaxes in the token stream. May 30 '19 at 11:57
• @StevenB.Segletes Mine works with {{{{{{}}}}}} now (and not by accident, like before :-) May 30 '19 at 12:52

The input for \numexpr ends when something (unexpandable) that can't appear in a \numexpr is found. Note that \numexpr triggers expansion until the input terminates as defined before.

If the token that signalled the end of the integer expression is \relax, it is removed altogether; thus it won't appear if you say

\edef\test{\the\numexpr1+1\relax}


which would expand to 2.

Braces are not allowed in integer expression, unless they're used for delimiting arguments to macros that are expanded as the integer expression is scanned. So

\def\addition#1#2{#1+#2}

will evaluate to 3. But \numexpr 1+{1+1}\relax is illegal, because the { stops the scanning and the operand for the first + is missing.
You can use ( and ) to delimit subexpressions to be evaluated with the usual precedence rules: \numexpr2*(1+3)\relax evaluates to 8.