Expandable test for an empty token list—methods, performance, and robustness

Question

With ε-TeX, the go-to method for testing if a <token-list> is empty is the following test:

\if\relax\detokenize{<token-list>}\relax
  % empty
\else
  % not empty
\fi

The method is fool-proof as long as the <token-list> can be safely \detokenized, which is the case when it is grabbed as argument to some other macro which does the testing.

Now looking at the expl3 sources I found the test to actually be (modulo _ and :)

\expandafter\ifx\expandafter\qnil\detokenize{#1}\qnil
  % empty
\else
  % not empty
\fi

where \qnil are “quarks” defined with \def\qnil{\qnil}, which means that \ifx\qnil<token> will only be true if <token> is \qnil, which will be the case only if #1 is empty; otherwise <token> will be any other (catcode-10 or 12) token which will make the test return false.

But this condition is also true for the first test: \if\relax<token> will only be true if <token> is another control sequence, which will never be the case if there's anything inside the \detokenize.

Or is it?

Is there a reason for the second method being preferred over the first? Is there an edge-case in which one of them would fail?

Both methods, as far as I can tell, apply the same treatment to the input token list, and are both robust regarding weird arguments, such as \iftrue\else\fi (which would otherwise be a problem) because in either case the <token-list> is \detokenized, so the argument can be virtually anything.

Motivation:

I’m working on some code that will use this test and should be executed a few hundred times for each function call, so performance is important. According to my tests the first method is slightly (very, very slightly) faster than the second:

\RequirePackage{l3benchmark}
\ExplSyntaxOn
\prg_new_conditional:Npnn \pho_tl_if_empty:n #1 { TF }
  {
    \if:w \scan_stop: \tl_to_str:n {#1} \scan_stop:
      \prg_return_true:
    \else:
      \prg_return_false:
    \fi:
  }
\cs_new:Npn \pho_test:N #1
  {
    \benchmark_tic:
    \int_step_inline:nn { 999999 }
      {
        #1 { } { } { } % Empty
        #1 { X } { } { } % non-empty
        #1 { \iftrue \else \fi } { } { } % just in case
      }
    \benchmark_toc:
  }
\pho_test:N \pho_tl_if_empty:nTF
\pho_test:N \tl_if_empty:nTF
\stop

output:

(l3benchmark) + TIC
(l3benchmark) + TOC: 2.17 s
(l3benchmark) + TIC
(l3benchmark) + TOC: 2.32 s

. . . Yes, those are 15 hundredths of a second in one million repetitions :-)

Thus, the motivation here is to know whether I can use the (in)significantly faster method without sacrificing robustness. The real motivation is to know in what way this type of choice may come to bite me in the future.

note that the quark-delimited test is older than \detokenize l3 is older than etex.... — David Carlisle, Oct 23, 2019 at 10:06
I tried benchmarking \if aa\fi versus \ifx aa\fi and the latter is slightly faster, but \expandafter\ifx aa\fi is noticebly slower. However, \if aa\fi is noticeably slower than \ifx\a\a\fi (with \def\a{a}) and \expandafter\ifx\aa\fi performs just slightly slower than \if aa\fi. — egreg, Oct 23, 2019 at 13:47
@siracusa Yes, I've already considered that. You are right, for longer arguments the \detokenize slows down the operation by a considerable amount. However for what I'm trying to do I need the \detokenize approach because it has to cope with possibly unbalanced conditionals in the argument, in which case other approaches all fail. Thanks for pointing it out! — Phelype Oleinik, Oct 23, 2019 at 14:36
If you really really really care about performance, don't use \prg_new_conditional:Npnn, but instead code the test yourself, because the way \prg_new_conditional:Npnn sets up the branching is slow (in the produced code, not during the definition, hint: it uses \expandafter). Instead, if you want the last tiny bit of performance you should use \cs_new:Npn \__pho_fi_use_i:wnn \fi: \use_ii:nn #1 #2 { \fi: #1 } \cs_new:Npn \pho_tl_if_empty:nTF #1 { \if:w \scan_stop: \tl_to_str:n { #1 } \scan_stop: \__pho_fi_use_i:wnn \fi: \use_ii:nn } — Skillmon, Nov 13, 2019 at 18:42

Skillmon · Accepted Answer · 2021-04-03 09:02:37Z

General

There are a few considerations when it comes to performance of TeX code:

argument grabbing costs time, don't grab arguments unnecessarily
\expandafter is slow, if you can work around it with the same amount of expansions it's faster, so instead of
```
\if...
  \expandafter\@firstoftwo
\else
  \expandafter\@secondoftwo
\fi
```
we'd use (this uses an aspect of the first point, too, namely if false only the contents of the true branch will be gobbled)
```
\long\def\my@fi@firstoftwo\fi#1#2#3{\fi#2}
\if...
  \my@fi@firstoftwo
\fi
\@secondoftwo
```
gobbling tokens explicitly as delimiters for arguments is faster than gobbling them as an argument which is delimited, so the above example can further be optimized:
```
\long\def\my@fi@firstoftwo\fi\@secondoftwo#1#2{\fi#1}
\if...
  \my@fi@firstoftwo
\fi
\@secondoftwo
```
But be aware that this way code becomes less readable, less reusable, and less maintainable, so the small performance gain comes at a cost.

\if... can represent any if test that results in a TeX-syntax if, such as \ifx AB, \iftrue, etc.

Also \if tests can be slow (depending on the used test) and so is \detokenize, if we can get around those, we should. Another thing to consider is that \if tests are not robust if their arguments contains other \if tests, \else or \fi. To overcome this the standard test for an empty argument does \detokenize the argument with:

\long\def\ifemptyStandard#1%
  {%
    \if\relax\detokenize{#1}\relax
      \expandafter\@firstoftwo
    \else
      \expandafter\@secondoftwo
    \fi
  }

This yields an unbeatable robustness, as the only possible argument that might fail this test would be an unbalanced input, which needs to be actively created, such as \expandafter\ifemptyStandard\expandafter{\iffalse{\fi}}{true}{false} (but who would do that anyway).

Of all the if tests built into TeX, \ifx is probably the fastest. So a naive test \ifx <some-token>#1<some-token> would be pretty fast, unfortunately this would not be robust. Cases for which it'd fail would be if \if..., \else, or \fi would be part of the argument or if #1 starts with <some-token> (though we can make <some-token> pretty unlikely).

Fast `\ifempty`

The following is a fast test, that considers some of the above mentioned aspects. We don't use any \if... test, but instead do the branching through TeX's argument grabbing logic:

\long\def\ifempty@true\ifempty@A\ifempty@B\@secondoftwo#1#2{#1}
\long\def\ifempty@#1\ifempty@A\ifempty@B{}
\long\def\ifempty#1%
  {%
    \ifempty@\ifempty@A#1\ifempty@B\ifempty@true
      \ifempty@A\ifempty@B\@secondoftwo
  }

So if #1 is empty \ifempty@ will gobble only the first \ifempty@A and \ifempty@B and \ifempty@true will be executed, gobbling the following \ifempty@A\ifempty@B\@secondoftwo and the false-branch. On the other hand, if #1 is not empty everything up to \@secondoftwo (non-inclusive) will be gobbled and \@secondoftwo will execute the false-branch.

This way we get a fast testing macro (taking about 70% the time of the \if\relax\detokenize{#1}\relax test during my benchmarks), that's fairly robust (only input which contains \ifempty@A\ifempty@B will fail the test, and that should be rare).

And of course, we can use tokens which are even more unlikely than \ifempty@A and \ifempty@B, e.g., why not use a <DEL> characters for both but with different category codes (that should be pretty very very unlikely to ever be part of a valid argument):

\begingroup
\lccode`\&=127
\lccode`\$=127
\catcode`\&=12
\catcode`\$=11
\lowercase{\endgroup
\long\def\ifempty@true&$\@secondoftwo#1#2{#1}
\long\def\ifempty@#1&${}
\long\def\ifempty#1{\ifempty@&#1$\ifempty@true&$\@secondoftwo}
}

Fast `\ifblank`

As a small addition, we can also create a fast \ifblank test based on the aforementioned thoughts. The standard \ifblank looks something like the following:

\long\def\ifblankStandard#1%
  {%
    \if\relax\detokenize\expandafter{\@gobble #1.}\relax
      \expandafter\@firstoftwo
    \else
      \expandafter\@secondoftwo
    \fi
  }

So essentially the same as \ifemptyStandard but with an \expandafter and a \@gobble #1. added. But we could do the same as for our fast \ifempty test with just some small additions (I'll just add this to the slightly obfuscated variant using the <DEL> tokens). And we don't want to use some \expandafters (remember they are slow) so we use \ifblank@ to gobble one token and insert the necessary tests of \ifempty.

\begingroup
\lccode`\&=127
\lccode`\$=127
\catcode`\&=12
\catcode`\$=11
\lowercase{\endgroup
\long\def\ifempty@true&$\@secondoftwo#1#2{#1}
\long\def\ifempty@#1&${}
\long\def\ifempty#1{\ifempty@&#1$\ifempty@true&$\@secondoftwo}
\long\def\ifblank@#1{\ifempty@&}
\long\def\ifblank#1{\ifblank@#1.$\ifempty@true&$\@secondoftwo}
}

Faster `\ifblank`

It is indeed possible to create an even faster \ifblank test, which is a bit less robust. The previous \ifblank would fail for a combination of tokens which have to be directly adjacent. This test fails if the argument contains a single marker. The test again uses TeX's argument grabbing logic, maybe in an even more ingenious way.

The speed advantage stems from the fact that TeX has to reinsert the first marker token (\ifempty@A) in the fast implementation if the argument isn't empty/blank. This implementation never needs to reinsert a token, instead it gobbles the first marker if #1 is blank, because it uses two parameters, one normal one and one delimited one. Also, it needs one step of expansion less.

The result is ca. 20% faster.

\long\def\ifblank#1%
  {%
    \ifblank@#1\ifblank@mark\ifblank@false
      \ifblank@mark\@firstoftwo
  }
\long\def\ifblank@#1#2\ifblank@mark{}
\long\def\ifblank@false\ifblank@mark\@firstoftwo#1#2{#2}

I'd argue about the unbalanced input, \expandafter\ifempty\expandafter{\iffalse{\fi}}. This is, for all effects, the same as \ifempty{}}, which doesn't make any sense. Other than that, excellent answer! — Phelype Oleinik, Nov 20, 2019 at 17:33
@PhelypeOleinik as I said, it has to be created malevolently. — Skillmon, Nov 20, 2019 at 18:27

Ulrich Diez · Accepted Answer · 2020-01-02 12:03:42Z

In case you need an expandable empty-test which does without e-TeX-extensions and without forbidden tokens, I can offer this one:

%%-----------------------------------------------------------------------------
%% Check whether argument is empty:
%%.............................................................................
%% \CheckWhetherEmpty{<Argument which is to be checked>}%
%%                   {<Tokens to be delivered in case that argument
%%                     which is to be checked is empty>}%
%%                   {<Tokens to be delivered in case that argument
%%                     which is to be checked is not empty>}%
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
%% <https://groups.google.com/forum/#!original/comp.text.tex/kuOEIQIrElc/lUg37FmhA74J>
%%
%% Due to \romannumeral0-expansion the result is delivered after two
%% expansion-steps/after two "hits" by \expandafter.
\long\def\firstoftwo#1#2{#1}%
\long\def\secondoftwo#1#2{#2}%
\long\def\CheckWhetherEmpty#1{%
  \romannumeral0\expandafter\secondoftwo\string{\expandafter
  \secondoftwo\expandafter{\expandafter{\string#1}\expandafter
  \secondoftwo\string}\expandafter\firstoftwo\expandafter{\expandafter
  \secondoftwo\string}\firstoftwo\expandafter{} \secondoftwo}%
  {\firstoftwo\expandafter{} \firstoftwo}%
}%

Like anything else that works in terms of macros, this does not work with arguments that contain \outer-tokens.

Deviating from the requirements formulated in the question, \CheckWhetherEmpty is rather slow.

I take \CheckWhetherEmpty for a moot thing/for a slow workaround in situations where one can't take for granted that e-TeX's \detokenize is available/is allowed by the terms of the macro-writing-challenge.

I emphasize that the gist/the basic idea of "hitting" either the first token of the non-empty argument or the closing brace behind the empty argument with \string and cranking out the brace-cases by removing a brace-balanced argument does not come from me but does come from Robert R. Schneck's \ifempty-macro.

I just added \romannumeral0-expansion and stringification and removal of superfluous curly braces via \secondoftwo in favor of removing superfluous curly braces via \iffalse..\fi.
I did so for ensuring that things won't break half-way through the expansion-chain due to unbalanced \if..\else..\fi at some stage popping up that might be contained in the argument or might come into being due to "hitting" the first token of the argument with \string...

In order to explain how the test works, let's rewrite this with different line-breaking:

\long\def\CheckWhetherEmpty#1{%
  \romannumeral0%
  \expandafter\secondoftwo\string{%
  \expandafter\secondoftwo % <- The interesting \secondoftwo
  \expandafter{% <- Opening brace of interesting \secondoftwo's first argument.
  \expandafter{%
  \string#1} % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
  \expandafter
  \secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
  \expandafter\firstoftwo\expandafter{\expandafter
  \secondoftwo\string}%
  \firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
  {\firstoftwo\expandafter{} \firstoftwo}%
}%

Let's look at the three scenarios:

Scenario 1: #1 is not empty and #1's first token is an opening brace—e.g., #1={foo}bar:

\CheckWhetherEmpty{{foo}bar}{empty}{not empty}%

Step 1:

\romannumeral0%
\expandafter\secondoftwo\string{%
\expandafter\secondoftwo % <- The interesting \secondoftwo
\expandafter{% <- Opening brace of interesting \secondoftwo's first argument.
\expandafter{%
\string{foo}bar} % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 2: \romannumeral0-expansion initiated:

%\romannumeral0-expansion in progress:
\expandafter\secondoftwo\string{%
\expandafter\secondoftwo % <- The interesting \secondoftwo
\expandafter{% <- Opening brace of interesting \secondoftwo's first argument.
\expandafter{%
\string{foo}bar} % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 3: \expandafter "hits" \string and { gets stringified:

%\romannumeral0-expansion in progress:
\secondoftwo{₁₂%
\expandafter\secondoftwo % <- The interesting \secondoftwo
\expandafter{% <- Opening brace of interesting \secondoftwo's first argument.
\expandafter{%
\string{foo}bar} % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 4: \secondoftwo removes {₁₂:

%\romannumeral0-expansion in progress:
\expandafter\secondoftwo % <- The interesting \secondoftwo
\expandafter{% <- Opening brace of interesting \secondoftwo's first argument.
\expandafter{%
\string{foo}bar} % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 5: \expandafter-chain "hits" \string which in case of the argument not being empty strigifies the argument's first token and in case of the argument being empty stringifies the closing brace:

%\romannumeral0-expansion in progress:
\secondoftwo % <- The interesting \secondoftwo
{% <- Opening brace of interesting \secondoftwo's first argument.
{%
{₁₂foo}bar} % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 6: The interesting \secondoftwo acts:

%\romannumeral0-expansion in progress:
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 7: \expandafter "hits" \string and } gets stringified:

%\romannumeral0-expansion in progress:
\secondoftwo}₁₂% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 8: \secondoftwo removes }₁₂:

%\romannumeral0-expansion in progress:
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 9: \expandafter-chain "hits" \string and } gets stringified:

%\romannumeral0-expansion in progress:
\firstoftwo{\secondoftwo}₁₂%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 10: \firstoftwo acts:

%\romannumeral0-expansion in progress:
\secondoftwo}₁₂%
\firstoftwo\expandafter{} \secondoftwo
{empty}{not empty}%

Step 11: \secondoftwo removes }₁₂:

%\romannumeral0-expansion in progress:
\firstoftwo\expandafter{} \secondoftwo
{empty}{not empty}%

Step 12: \firstoftwo acts:

%\romannumeral0-expansion in progress:
\expandafter⟨space token⟩\secondoftwo
{empty}{not empty}%

Step 13: \expandafter "hits" \secondoftwo:

%\romannumeral0-expansion in progress:
⟨space token⟩not empty%

Step 14: \romannumeral0-expansion finds the ⟨space token⟩ and discards it and stops searching for more digits. Thus \romannumeral finds the non-positive number 0 and therefore terminates without delivering any token in return:

%\romannumeral0-expansion terminated:
not empty%

Scenario 2: #1 is not empty and #1's first token is not an opening brace—e.g., #1=foobar:

\CheckWhetherEmpty{foobar}{empty}{not empty}%

Step 1:

\romannumeral0%
\expandafter\secondoftwo\string{%
\expandafter\secondoftwo % <- The interesting \secondoftwo
\expandafter{% <- Opening brace of interesting \secondoftwo's first argument.
\expandafter{%
\string foobar} % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 2: \romannumeral0-expansion initiated:

%\romannumeral0-expansion in progress:
\expandafter\secondoftwo\string{%
\expandafter\secondoftwo % <- The interesting \secondoftwo
\expandafter{% <- Opening brace of interesting \secondoftwo's first argument.
\expandafter{%
\string foobar} % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 3: \expandafter "hits" \string and { gets stringified:

%\romannumeral0-expansion in progress:
\secondoftwo{₁₂%
\expandafter\secondoftwo % <- The interesting \secondoftwo
\expandafter{% <- Opening brace of interesting \secondoftwo's first argument.
\expandafter{%
\string foobar} % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 4: \secondoftwo removes {₁₂:

%\romannumeral0-expansion in progress:
\expandafter\secondoftwo % <- The interesting \secondoftwo
\expandafter{% <- Opening brace of interesting \secondoftwo's first argument.
\expandafter{%
\string foobar} % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 5: \expandafter-chain "hits" \string which in case of the argument not being empty strigifies the argument's first token and in case of the argument being empty stringifies the closing brace:

%\romannumeral0-expansion in progress:
\secondoftwo % <- The interesting \secondoftwo
{% <- Opening brace of interesting \secondoftwo's first argument.
{%
f₁₂oobar} % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 6: The interesting \secondoftwo acts:

%\romannumeral0-expansion in progress:
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 7: \expandafter-chain "hits" \string and } gets stringified::

%\romannumeral0-expansion in progress:
\firstoftwo{\secondoftwo}₁₂%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 8: \firstoftwo acts:

%\romannumeral0-expansion in progress:
\secondoftwo}₁₂%
\firstoftwo\expandafter{} \secondoftwo
{empty}{not empty}%

Step 9: \secondoftwo removes }₁₂:

%\romannumeral0-expansion in progress:
\firstoftwo\expandafter{} \secondoftwo
{empty}{not empty}%

Step 10: \firstoftwo acts:

%\romannumeral0-expansion in progress:
\expandafter⟨space token⟩\secondoftwo
{empty}{not empty}%

Step 11: \expandafter "hits" \secondoftwo:

%\romannumeral0-expansion in progress:
⟨space token⟩not empty%

Step 12: \romannumeral0-expansion finds the ⟨space token⟩ and discards it and stops searching for more digits. Thus \romannumeral finds the non-positive number 0 and therefore terminates without delivering any token in return:

%\romannumeral0-expansion terminated:
not empty%

Scenario 3: #1 is empty:

\CheckWhetherEmpty{}{empty}{not empty}%

Step 1:

\romannumeral0%
\expandafter\secondoftwo\string{%
\expandafter\secondoftwo % <- The interesting \secondoftwo
\expandafter{% <- Opening brace of interesting \secondoftwo's first argument.
\expandafter{%
\string} % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 2: \romannumeral0-expansion initiated:

%\romannumeral0-expansion in progress:
\expandafter\secondoftwo\string{%
\expandafter\secondoftwo % <- The interesting \secondoftwo
\expandafter{% <- Opening brace of interesting \secondoftwo's first argument.
\expandafter{%
\string} % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 3: \expandafter "hits" \string and { gets stringified:

%\romannumeral0-expansion in progress:
\secondoftwo{₁₂%
\expandafter\secondoftwo % <- The interesting \secondoftwo
\expandafter{% <- Opening brace of interesting \secondoftwo's first argument.
\expandafter{%
\string} % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 4: \secondoftwo removes {₁₂:

%\romannumeral0-expansion in progress:
\expandafter\secondoftwo % <- The interesting \secondoftwo
\expandafter{% <- Opening brace of interesting \secondoftwo's first argument.
\expandafter{%
\string} % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 5: \expandafter-chain "hits" \string which in case of the argument not being empty strigifies the argument's first token and in case of the argument being empty stringifies the closing brace:

%\romannumeral0-expansion in progress:
\secondoftwo % <- The interesting \secondoftwo
{% <- Opening brace of interesting \secondoftwo's first argument.
{%
}₁₂ % <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is an opening brace (Scenario 1).
\expandafter
\secondoftwo\string}% <- Closing brace of interesting \secondoftwo's first argument in case #1's first token is not an opening brace (Scenario 2).
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}%
\firstoftwo\expandafter{} \secondoftwo}% <- Closing brace of interesting \secondoftwo's first argument in case #1 is empty (Scenario 3).
{\firstoftwo\expandafter{} \firstoftwo}%
{empty}{not empty}%

Step 6: The interesting \secondoftwo acts:

%\romannumeral0-expansion in progress:
\firstoftwo\expandafter{} \firstoftwo
{empty}{not empty}%

Step 7: \firstoftwo acts:

%\romannumeral0-expansion in progress:
\expandafter⟨space token⟩\firstoftwo
{empty}{not empty}%

Step 8: \expandafter "hits" \firstoftwo:

%\romannumeral0-expansion in progress:
⟨space token⟩empty%

Step 9: \romannumeral0-expansion finds the ⟨space token⟩ and discards it and stops searching for more digits. Thus \romannumeral finds the non-positive number 0 and therefore terminates without delivering any token in return:

%\romannumeral0-expansion terminated:
empty%

Based on that you can implement an \ifblank-test as follows:

%%-----------------------------------------------------------------------------
%% Check whether argument is blank (empty or only spaces):
%%-----------------------------------------------------------------------------
%% -- Take advantage of the fact that TeX discards space tokens when
%%    "fetching" _un_delimited arguments: --
%% \CheckWhetherBlank{<Argument which is to be checked>}%
%%                   {<Tokens to be delivered in case that
%%                     argument which is to be checked is blank>}%
%%                   {<Tokens to be delivered in case that argument
%%                     which is to be checked is not blank}%
\long\def\CheckWhetherBlank#1{%
  \romannumeral\expandafter\expandafter\expandafter\secondoftwo
  \expandafter\CheckWhetherEmpty\expandafter{\firstoftwo#1{}.}%
}%

Based on the gist of the implementation of \CheckWhetherEmpty you can implement checking whether a non-delimited argument's first token is an explicit character token of category code 1 (begin group): Just ensure by appending a dot that the \string which gets carried out right before executing the "interesting \secondoftwo" never "hits" a closing brace (which implies elimination of scenario 3) and implement forking between scenario 1 and scenario 2:

%%-----------------------------------------------------------------------------
%% Check whether argument's first token is a catcode-1-character
%%-----------------------------------------------------------------------------
%% \CheckWhetherBrace{<Argument which is to be checked>}%
%%                   {<Tokens to be delivered in case that argument
%%                     which is to be checked has leading
%%                     catcode-1-token>}%
%%                   {<Tokens to be delivered in case that argument
%%                      which is to be checked has no leading
%%                      catcode-1-token>}%
%%
%% Due to \romannumeral0-expansion the result is delivered after two
%% expansion-steps/after two "hits" by \expandafter.
%%
\long\def\CheckWhetherBrace#1{%
  \romannumeral0\expandafter\secondoftwo\expandafter{\expandafter{%
  \string#1.}\expandafter\firstoftwo\expandafter{\expandafter
  \secondoftwo\string}\firstoftwo\expandafter{} \firstoftwo}%
  {\firstoftwo\expandafter{} \secondoftwo}%
}%

Thanks a lot for the thorough explanation! I remember dissecting your \CheckWhetherNull (that was the name at the time) macro a few months ago when adapting some code for this package, to find out precisely what it does. It's indeed a masterful expansion management (and quite hard to come up with). Though its increased robustness comes at the price of it taking twice as long to run (in my test above) compared to the \detokenize approach, so I won't use it this time. But thanks again for the thorough explanation! — Phelype Oleinik, Jan 1, 2020 at 23:53
@UlrichDiez please note that using \romannumeral with a 0 isn't as robust as using \romannumeral`\^^@ (which also is faster). — Skillmon, Oct 28, 2020 at 14:19
@Skillmon Yes, \romannumeral`\^^@ is faster. But why is \romannumeral0 not as robust as \romannumeral`\^^@? Do you have things like \upper-/\lowercase in mind in situations where 0 has some lccode/uccode assigned which may turn it into some non-zero? — Ulrich Diez, Oct 28, 2020 at 18:00
@UlrichDiez when another number follows the 0 it is considered part of the \romannumeral, so \romannumeral0\empty3 will result in iii, whereas \romannumeral`\^^@\empty3 will result in 3. In (almost?) all situations in which \romannumeral is used to trigger expansion, the second result is the one you want to get. — Skillmon, Oct 28, 2020 at 19:31
@Skillmon When TeX is gathering the <number>-quantity for \romannumeral and has found the 0, TeX looks, hereby expanding expandable tokens, for the presence of subsequent <digit>s or of an <optional space> (where only the catcode does matter, not the character code) which in any case terminates the number. Therefore I always (unless when producing a bug in my code) ensure that the expansion-cascade after \romannumeral0 yields a token-sequence whose very first token is that <optional space>. That <optional space> gets discarded and terminates the search for more digits. — Ulrich Diez, Oct 28, 2020 at 20:03

Stack Exchange Network

current community

your communities

more stack exchange communities

Expandable test for an empty token list—methods, performance, and robustness

Or is it?

Motivation:

2 Answers 2

General

Fast `\ifempty`

Fast `\ifblank`

Faster `\ifblank`

Your Answer

Not the answer you're looking for? Browse other questions tagged conditionals expl3 token-lists performance e-tex or ask your own question.

Linked

Hot Network Questions

Expandable test for an empty token list—methods, performance, and robustness

Or is it?

Motivation:

2 Answers 2

General

Fast \ifempty

Fast \ifblank

Faster \ifblank

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged conditionals expl3 token-lists performance e-tex or ask your own question.

Linked

Related

Hot Network Questions

Fast `\ifempty`

Fast `\ifblank`

Faster `\ifblank`