32

Is it possible to test whether a token register is empty without expanding it?

7 Answers 7

33

At time of writing the other TeX based answers on this page are flawed in that they hide a conditional \ifx inside a macro but still use \else/\fi at the "top" level. This will mean that things break unexpectedly when used inside other conditionals.

The LaTeX3 programming language expl3 contains a module for doing stuff with token registers:

\usepackage{expl3}
...
\ExplSyntaxOn
\toks_if_empty:NTF \mytoks {true} {false}
\ExplSyntaxOff

It essentially does internally what the other answers here are suggesting, but it uses expansion to grab its arguments so the branching is robust (and you don't have \fi lying around to get in your way).

Update: So what does this approach do that is superior to other methods? Consider the style of solution first offered in answer to this question:

\def\IfEmpty#1{%
  \edef\1{\the#1}
  \ifx\1\empty
}
...
\IfEmpty\foo T\else F\fi

This doesn't behave nicely when nested, because TeX scans ahead when discarding unfollowed branches of a conditional. Consider

\ifx\bar\baz
  \IfEmpty\foo T\else F\fi % <- uh oh
\else
  E
\fi

If \bar = \baz, then the second branch is discarded and the first branch is executed. So far so good. If \bar\baz, then the first branch is discarded by reading ahead until the first unmatched \else — and this is the one in the line labelled "uh oh" above. So you could collapse the expansion of the above snippet in this case to:

\iffalse\else F\fi % <- uh oh
\else
  E
\fi

and hence the cause of the ‘Extra \else’ error message in this case.

So this form for conditionals doesn't work so well. Next try. You can also write this style of code like this:

\def\IfEmpty#1#2#3{%
  \edef\1{\the#1}
  \ifx\1\empty
    #2%
  \else
    #3%
  \fi
}

This avoids the problems of nesting as in the previous trial solution, but it's prone to another problem: #2 and #3 have trailing material behind them, namely \else and \fi. This is a problem if you want to write something like

\def\processfoo#1{...something with #1...}
\IfEmpty\foo{\error}{\processfoo} {arg}

because the #1 passed to \processfoo will be \fi instead of the desired {arg}. The conditional in this case is better written as

\def\IfEmpty#1#2#3{%
  \edef\1{\the#1}
  \ifx\1\empty
    \expandafter\@firstoftwo
  \else
    \expandafter\@secondoftwo
  \fi
  {#2}{#3}
}

so overcome this problem. This is how expl3 conditionals work, and it's why we're writing TF at the end of all their names to indicate "true" and "false" branches. (Or just T or just F if you only want one of them.)

Incidentally, there are expandable tests for checking for emptiness, which is why I suggest using the expl3 approach for this test. Expandability is not always required, of course, but code that is fully expandable tends to be more reliable and it's always nice to have for cases such as

\typeout{ \toks_if_empty:NT \foo {Warning:~\string\foo\space is~ empty} }
10
  • Did my updated answer fix the problems? If so, is there a way to do it that doesn't involves a sentinel?
    – TH.
    Sep 10, 2010 at 5:42
  • Yes, it did. That's essentially the definition of the expl3 version, although the "emptiness" test in expl3 is slightly more robust. I'm not exactly sure what you mean specifically and exactly by "sentinel", but I think the answer is no. Sep 10, 2010 at 7:06
  • 2
    I like such examples in expl3 syntax (and also the one in luatex) very much. It help to get used to their look and feel. E.g. I was quite confused about the \space behind \foo. The idea that \foo swallows a ~ like it swallows normally a space is - well - curious. Sep 10, 2010 at 8:11
  • 1
    @Ulrike: inside \ExplSyntaxOn, tilde has the catcode of a space because space has the catcode of "ignored". If that's not to your liking, you can use \ExplSyntaxNamesOn, which is the same as \ExplSyntaxOn but leaves spaces as-is. We should perhaps be promoting this more for use in general package code, since I guess the space-ignore-rule is probably somewhat inconvenient for integrating some expl3 code into regular LaTeX code. Sep 10, 2010 at 10:33
  • 3
    @TH. There is a way which doesn't involve a sentinel (sort of), using \detokenize. It begins like \expandafter\ifx\expandafter\notsentinel\detokenize\expandafter{\the#1}\notsentinel. Here, \notsentinel is used as a sentinel, but thanks to \detokenize there is not risk that it accidentally matches the contents of the toks register tested.
    – mpg
    Oct 28, 2010 at 13:44
24

With LuaTeX you can do this:

\directlua{
if string.len(tex.toks["headline"]) > 0 then
    print("not emtpy")
end
}

But since I never use token list: don't use this code to build nuclear plants.

4
  • I'm curious why this got a down vote, does it not work?
    – TH.
    Sep 10, 2010 at 5:43
  • If it doesn't work, I'd like to get a comment about what is wrong. My tests were all okay.
    – topskip
    Sep 10, 2010 at 7:02
  • 3
    I was rash. Using Lua to solve a TeX problem seems a bit batty to me (as in, why not skip using toks entirely and just do everything in Lua?). But at least the Lua code here is nice and straightforward, unlike the TeX code required. Sep 10, 2010 at 7:02
  • 10
    We definitely need more answers of this kind. Even if it is solvable in TeX, a solution that mere humans can understand is highly desirable. (heck, even I understood it, unlike the other black expansion magic). Sep 10, 2010 at 7:20
15

Ulrich Diez regularly posts on comp.text.tex some code along the following lines (slightly modified by me for toks):

\newcommand\@ifempty@toks[1]{%
  \ifcat\relax\detokenize\expandafter{\the#1}\relax
    \expandafter\@firstoftwo
  \else
    \expandafter\@secondoftwo
  \fi
}

Or, without using e-TeX,

\newcommand\@ifempty@toks[1]{%
  \ifcase\iffalse{{{\fi\expandafter\@ifempty@@toks\the#1}1}1}0
    \expandafter\@firstoftwo
  \else
    \expandafter\@secondoftwo
  \fi}
\newcommand{\@ifempty@@toks}
  {\expandafter\@gobble\expandafter{\expandafter{%
        \ifcase`}\expandafter}\expandafter\fi\string}

EDIT: Ulrich Diez rightfully points out that redefining \relax can make the first test fail. It is slightly safer to use a character token like $ or X that has category code among 3,4,7,8,11.

Additionally, neither method works if the \toks contains outer macros, as in \outer\def\foo{} \newtoks\mytoks \mytoks=\expandafter{\noexpand\foo}.

9
  • The non e-TeX version is an excellent example of using \ifcase to 'bury' the extra braces. It is missing two \expandafter's however: one before '`' and one after it (in the definition of \@ifempty@@toks). I think it would also be nice if standard \def' were used instead of \LaTeX lingo. Thanks for the idea, anyway!
    – alexsh
    Mar 10, 2014 at 2:56
  • Another comment is that to make it possible to use this macro inside alignments, `} should be replaced by 1\expandafter} (see The TeXbook, p. 385 for the reasons and the discussion of the balance and master counters). In any case, this is a beautiful macro!
    – alexsh
    Mar 10, 2014 at 18:50
  • 1
    @alexsh The two \expandafter you suggest adding would do nothing. When TeX looks for a number after \ifcase, it reads the next two tokens, which denote the character code of }. It turns out that such a number can be followed by an optional space token, so TeX continues expanding what follows (namely \expandafter}\expandafter...) before using the number (character code of }) as the argument of \ifcase. Mar 12, 2014 at 13:25
  • As far as I can tell without testing, your second comment is correct: to make the macro useable inside alignments, one should write \ifcase1\expandafter}\expandafter}\expandafter\fi. Mar 12, 2014 at 13:27
  • 1
    Instead of \ifcat\relax\detokenize\expandafter{\the#1}\relax I suggest \ifcat$\detokenize\expandafter{\the#1}$ in order not to rely on \relax not being redefined to fool the test as would be the case, e.g., with \def\relax{ZZ}. (Instead of $ you can use some other character token not of category 12(other) or 10(space) or 13(active). ) A drawback of all these approaches is processing the result of \the<token-register> by means of macros: This fails if the token-register contains tokens defined in terms of \outer (after placing them into the register). Jan 10 at 13:21
9

Would something like this work?

\def\ifempty#1{%
\edef\ContentsOfList{\the#1}%
\edef\CompareTo{}%
\ifx\ContentsOfList\CompareTo}

You could then use it like this:

\toks0={\blah\bloh a b c}
\toks2={}

\ifempty{\toks0} empty\else nonempty\fi

\ifempty{\toks2} empty\else nonempty\fi
10
  • And \the inside \edef inserts the token list without expanding it. Yes, it works, I should have come up with that by myself. Sep 9, 2010 at 17:59
  • 1
    This does have the disadvantage that it isn't expandable. I.e., you couldn't use it like \edef\foo{\ifempty{\toks0} \foo\else\bar}
    – TH.
    Sep 9, 2010 at 18:02
  • @TH: To solve this without assignment do something like \expandafter\helper\the#1 \sentinel\anothersentinel and then make \helper do whatever it takes, namely pick something out of the expanded #1. This probably means \helper has to expand to \expandafter\anotherhelper\string, and now it's getting complicated. And perhaps can't be reliably done, however clever we are with sentinel. Sep 9, 2010 at 20:50
  • @Jan: We expect \ifx\a\b \ifempty{#1} \fi \fi to expand to nothing if the \ifx is false. But your definition does not have this property. Sep 9, 2010 at 20:56
  • Continuing JF's last comment, I do not recommend defining conditionals in this way. It will blow up in your face when you least expect it, usually when you try and nest this conditional within another. Sep 9, 2010 at 23:03
6

Similar to Jan Hlavacek's answer, but expandable:

\def\certainlynotintoks{\certainlynotintoks}
\def\iftoksempty#1{\expandafter\ifx\expandafter
    \certainlynotintoks\the#1 \certainlynotintoks}

\toks0{\undefined will not be expanded}
\toks2{}
\iftoksempty{\toks0} empty\else not empty\fi \par
\iftoksempty{\toks2} empty\else not empty\fi

This requires that \certainlynotintoks does not appear as the first token in the token register. Both this and Jan's answer do expand the token register, but only once. I don't believe there's a way to avoid that.

Edit: Taking Will's (absolutely correct) comment into consideration, change the definition of \iftoksempty to

\makeatletter
\def\iftoksempty#1{\expandafter\ifx\expandafter
    \certainlynotintoks\the#1 \certainlynotintoks \expandafter\@firstoftwo
    \else \expandafter\@secondoftwo\fi}

Then you can use \iftoksempty{\toks0}{empty}{not empty}.

5
  • In fact, \the\toksreg simply inserts the token list without any expansion at all. Sep 9, 2010 at 18:04
  • Yes, that's what I meant by expansion. \the\toksreg expands to whatever that token register contains.
    – TH.
    Sep 9, 2010 at 18:06
  • Again, hiding \ifx in a macro has dire consequences. It breaks nesting because TeX can't match its trailing \else...\fi with a leading \iftrue/\iffalse-equivalent conditional. Sep 9, 2010 at 23:07
  • Yes, you're correct, of course.
    – TH.
    Sep 10, 2010 at 4:10
  • What about the token-register in question containing unbalanced/unmatched \if or \else or \fi? Jan 10 at 13:27
5

A rather retro way of avoiding the problems Will talks about is to use flags. Define:

\newif\iftest 
\def\testtoksempty#1{\edef\1{\the#1}
  \ifx \1\empty \testtrue % from plain.tex: \def\empty{}
  \else\testfalse \fi}

Then use:

\testtoksempty\toksreg
\iftest toksreg is empty \else toksreg's got something \fi

I tend to avoid any kind of overlap between \if conditional and macro expansion because of my incomplete, fearful grasp of the issues Will nicely explained (and which I now understand a bit better).
This is clunky, ASM-like way to code, and the use of a fixed test flag means that Boolean expressions are going to be fiddly: you may be driven to multiple, nearly identical macros - \tessttoksempty, \tesssttoksempty, etc.
But it's robust and easy to debug.

0
1

With all "traditional" approaches presented so far and with expl3's \tl_if_empty:VTF the content of the token-register is produced via \the and examined by means of macros.

The circumstance of a macro-argument containing \outer-tokens triggers an error-message.

Therefore these "taditional" approaches and expl3's \tl_if_empty:VTF fail if the register to test for emptiness contains tokens that at the time when being produced by \the are defined in terms of \outer.

E.g., with

\newtoks\mytoks
\mytoks{\foo}%
\outer\def\foo{foo is outer}%

\the\mytoks will produce a token which is defined in terms of \outer and therefore cannot be a component of a macro-argument.

(However, interface3.pdf, the reference documentation for the expl3 programming environment, explicitly states in section "1.4 TeX concepts not supported by LaTeX3":

The TeX concept of an "\outer" macro is not supported at all by LaTeX3. As such, the functions provided here may break when used on top of LaTeX 2ε if \outer tokens are used in the arguments.

)

If working on a LuaLaTeX-engine, according to LuaTeX Reference Manual stable December 2021 Version 1.15, you can do something like this, where \outer-tokens are not a problem:

\errorcontextlines=10000
\makeatletter
%%=============================================================================
\@ifdefinable\UD@stopromannumeral{\chardef\UD@stopromannumeral=`\^^00}%
\@ifdefinable\gobbletoks{%
  \begingroup
  \escapechar=-1
  \expandafter\endgroup\expandafter\def\expandafter\gobbletoks\expandafter#\expandafter1\string\toks{}%
}%
\newcommand\toksnum[1]{\expandafter\gobbletoks\meaning#1}%

% Don't use tex.sprint or the like for sending control-sequence-tokens to TeX-level
% as tex.sprint("\\macro") only works out as long as backslash has catcode 0.
\newcommand\CheckWhetherTokenRegisterEmpty[1]{%
  \romannumeral
  \directlua{%
    if string.len(tex.toks[\number\toksnum#1]) > 0 then
      token.put_next ( token.create("@secondoftwo") ) else token.put_next ( token.create("@firstoftwo") )
    end
    token.put_next ( token.create("expandafter"), token.create("UD@stopromannumeral") )
  }%
}%
%%=============================================================================
\makeatother

\newtoks\testtoks

\testtoks={}
\message{%
  ^^J%
  \string\testtoks
  \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
}%

\testtoks={ }
\message{%
  ^^J%
  \string\testtoks
  \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
}%

\testtoks={something}
\message{%
  ^^J%
  \string\testtoks
  \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
}%

% This is not a problem:

\testtoks={\foobar}
\outer\def\foobar{foobar is outer}
\message{%
  ^^J%
  \string\testtoks
  \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
}%

\catcode`\/=0 \catcode`\\=12
/message{%
  ^^J%
  /string/testtoks
  /CheckWhetherTokenRegisterEmpty/testtoks{ is}{ is not} empty%
}%
/catcode`/\=0 \catcode`//=12

\stop

If Lua-extensions are not available and if you can rely on the content of the token-register not containing \outer-tokens you can probably do something like this:

\errorcontextlines=10000
\makeatletter
%%=============================================================================
%% PARAPHERNALIA:
%% \UD@firstoftwo, \UD@secondoftwo,\UD@stopromannumeral, \UD@CheckWhetherNull,
%%=============================================================================
\newcommand\UD@firstoftwo[2]{#1}%
\newcommand\UD@secondoftwo[2]{#2}%
\@ifdefinable\UD@stopromannumeral{\chardef\UD@stopromannumeral=`\^^00}%
%%-----------------------------------------------------------------------------
%% Check whether argument is empty:
%%.............................................................................
%% \UD@CheckWhetherNull{<Argument which is to be checked>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is empty>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is not empty>}%
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
%% <https://groups.google.com/forum/#!original/comp.text.tex/kuOEIQIrElc/lUg37FmhA74J>
\newcommand\UD@CheckWhetherNull[1]{%
  \romannumeral\expandafter\UD@secondoftwo\string{\expandafter
  \UD@secondoftwo\expandafter{\expandafter{\string#1}\expandafter
  \UD@secondoftwo\string}\expandafter\UD@firstoftwo\expandafter{\expandafter
  \UD@secondoftwo\string}\expandafter\UD@stopromannumeral\UD@secondoftwo}{%
  \expandafter\UD@stopromannumeral\UD@firstoftwo}%
}%
%%-----------------------------------------------------------------------------
%% With eTeX do something like:
%% 
%% \newcommand\UD@CheckWhetherNull[1]{%
%%   \romannumeral\ifcat$\detokenize{#1}$%
%%     \expandafter\expandafter\expandafter\UD@stopromannumeral
%%     \expandafter\UD@firstoftwo
%%   \else
%%     \expandafter\expandafter\expandafter\UD@stopromannumeral
%%     \expandafter\UD@secondoftwo
%%   \fi
%% }%
%%-----------------------------------------------------------------------------
%% Check whether token-register is empty:
%%.............................................................................
\newcommand\CheckWhetherTokenRegisterEmpty[1]{%
  \romannumeral
  \expandafter\expandafter\expandafter\UD@secondoftwo
  \expandafter\UD@CheckWhetherNull
  \expandafter{\the#1}%
}%
%%=============================================================================
\makeatother

\newtoks\testtoks

\testtoks={}
\message{%
  ^^J%
  \string\testtoks
  \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
}%

\testtoks={ }
\message{%
  ^^J%
  \string\testtoks
  \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
}%

\testtoks={something}
\message{%
  ^^J%
  \string\testtoks
  \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
}%

% This is a problem:
%
% \testtoks={\foobar}
% \outer\def\foobar{foobar is outer}
% \message{%
%   ^^J%
%   \string\testtoks
%   \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
% }%

\stop

Another interesting question might be the handling of the case of the argument of the routine for checking emptiness of a token-register not denoting a token-register at all.

But the question of how to implement a test for checking if an arbitrary given argument denotes a token-register is a different question and thus is out of the scope of this question.

Still, a few thoughts on this that get one used to the idea that TeX may not allow you to implement every testing-routine one hundred percent accurately and reliably:

In case of the argument denoting a token-register the argument might consist of

  • a single \toksdef-token,
  • \toks⟨s.th. that expands to a TeX-⟨number⟩-quantity with a value in range of available token-registers⟩.

So a check would imply handling the case of the first token of the argument being the \toks-primitive in which case you would need to check if (recursive) expansion of following tokens yields only a valid TeX-⟨number⟩-quantity. As following tokens might form an expansion-based algorithm, the check would imply the task of checking if an arbitrary expansion-based algorithm terminates at all, and if so, if termination goes along with only delivering a valid TeX-⟨number⟩-quantity. Checking if an arbitrary expansion-based algorithm terminates at all reminds me of the halting-problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.