# Different command behavior if argument is not a single token

In defining a new LaTeX command, I'm looking for a way to distinguish whether its argument is a single token or not. In other words, I would like \foo x to be equivalent to \something{x}, and \foo{xy} to \somethingelse{xy} (as opposed to \something{xy}). Would that be possible somehow, possibly without using evil hacks?

I tried looking at the documentation for xparse and etoolbox, but I could not find anything useful. Any hints?

expl3 has a conditional just for that: \tl_if_single_token:nTF. It will return true only if the argument is exactly a single token. It returns false if the argument is empty, or if the argument is a braced group containing any number of tokens. expl3 also has \tl_if_single:nTF that returns true on x as well as on {x}, whereas \tl_if_single_token:nTF returns false on the latter.

\documentclass{article}

\ExplSyntaxOn
\cs_new_eq:NN \IfSingleTokenTF \tl_if_single_token:nTF
\cs_new_eq:NN \IfSingleItemTF \tl_if_single:nTF
\ExplSyntaxOff

\newcommand\foo[1]{%
\texttt{>#1: }%
\IfSingleTokenTF{#1}%
{\something}%
{\somethingelse}{#1}}

\newcommand\something[1]{(single:#1)}
\newcommand\somethingelse[1]{(multiple:#1)}

\begin{document}
\foo{}

\foo{ }

\foo{x}

\foo{ x}

\foo{xy}

\foo{{xy}}
\end{document}


As Frank noted in the comment, the code above works fine for XeTeX and LuaTeX because those engines tread UTF-8 characters like ä or 🦤 as a single token. pdfTeX, on the other hand, treats ä as two and 🦤 as four tokens, which when typesetting combine to produce a single glyph, thus the test above results as multiple tokens.

Depending on your use case (or if you want cross-engine compatibility) you might want to treat UTF-8 characters as a single token, in which case a bit more code is needed to check for those cases in pdfTeX:

The code first does the \tl_if_single_token:nTF test to eliminate the obvious case that the argument is indeed a single token. If that returns false, then we look at meaning of the first token in the argument, and if it is \UTFviii@(two|three|four)@octets then we know the first token is an UTF-8 char, in which case we remove the remaining tokens that compose the char and then test if what remains is empty: if it is, the only thing in the argument was the UTF-8 char, so return true, otherwise return false.

\documentclass{article}

\ExplSyntaxOn
\bool_lazy_or:nnTF
{ \sys_if_engine_luatex_p: }
{ \sys_if_engine_xetex_p: }
{
\cs_new_eq:NN \IfSingleTokenTF \tl_if_single_token:nTF
\use_none:n
}
{ \makeatletter \use:n }
{
\makeatother
\scan_new:N \s__fudo
\prg_new_conditional:Npnn \fudo_if_single_token:n #1 { T, F, TF }
{
\tl_if_single_token:nTF {#1}
{ \prg_return_true: }
{
{
\use:e
{
\exp_not:N \__fudo_if_single_token_aux:w
\exp_not:o { \token_to_meaning:N #1 }
\tl_to_str:n { UTFviii@ one @octets } ~ \s__fudo
\exp_not:n {#1} \s__fudo
}
}
{ \prg_return_false: }
}
}
\use:e
{
\cs_new:Npn \exp_not:N \__fudo_if_single_token_aux:w
#1 \tl_to_str:n { UTFviii@ } #2 \tl_to_str:n { @octets } ~
#3 \s__fudo #4 \s__fudo
}
{
\str_case:nnTF {#2}
{
{ one   } { \exp_after:wN \__fudo_single_chk:w \use_none:n }
{ two   } { \exp_after:wN \__fudo_single_chk:w \use_none:nn }
{ three } { \exp_after:wN \__fudo_single_chk:w \use_none:nnn }
{ four  } { \exp_after:wN \__fudo_single_chk:w \use_none:nnnn }
}
{ #4 \s__fudo }
{ \prg_return_false: }
}
\cs_new:Npn \__fudo_single_chk:w #1 \s__fudo
{
\tl_if_empty:nTF {#1}
{ \prg_return_true: }
{ \prg_return_false: }
}
\cs_new_eq:NN \IfSingleTokenTF \fudo_if_single_token:nTF
}
\ExplSyntaxOff

\newcommand\foo[1]{%
\texttt{>#1: }%
\IfSingleTokenTF{#1}%
{\something}%
{\somethingelse}{#1}}

\newcommand\something[1]{(single:#1)}
\newcommand\somethingelse[1]{(multiple:#1)}

\begin{document}
\foo{}

\foo{ }

\foo{x}

\foo{ä}

% \foo{鸟}

% \foo{🦤}

\foo{ x}

\foo{xy}

\foo{{xy}}
\end{document}


The code does not understand é (U+0065 Latin Small Letter E + U+0301 Combining Acute Accent) as a single character. You have to input the proper Unicode U+00E9 Latin Small Letter E with Acute.

• @fudo You can add %%begin novalidate before the expl3 code, and %%end novalidate after it so that Overleaf won't check that bit of code – Phelype Oleinik May 11 at 14:41
• @PhelypeOleinik small problem Phelype: \foo{ä} is single in XeTeX/LuaTeX but multiple in pdfTeX. While correct, perhaps not what is wanted, a user facing 2e command should probably detect multi-byte chars (and LICRs?) – Frank Mittelbach May 11 at 20:17
• @fudo Depending on your use-case, you might want to take a look at the new version I added – Phelype Oleinik May 12 at 14:42
• @JosephWright Yeah, maybe it would be useful. Something like \tl function but \text: \text_if_single_char:nTF, and \text_if_head_is_N_type:nTF and so on (\text_reverse:n will be fun :) – Phelype Oleinik May 12 at 14:44
• @fudo Don't worry ;-) I imagined you would need the single item version: checking for a single token is a really specialised, most likely useful only for programming tasks (and even then, rarely) – Phelype Oleinik May 13 at 12:18

EDITED to make it expandable! (see SUPPLEMENT)

\documentclass{article}
\newcommand\foo[1]{\fooaux#1\endfoo}
\def\fooaux#1#2\endfoo{\if\relax\detokenize{#2}\relax
\expandafter\fooproper\else\expandafter\fooalt\fi{#1#2}}
\newcommand\fooproper[1]{FOO! #1}
\newcommand\fooalt[1]{bar #1}
\begin{document}
\foo a

\foo{A}

\edef\z{\foo{ab}}% EXPANDABLE

\z
\end{document}


SUPPLEMENT

Ulrich rightly points out that the above simple solution has several potential chinks, depending on what the OP expects to get thrown at the \foo macro. In particular, if the argument is a delimited space, or contains a leading space as part of a delimited argument, the space will not be detected. Further, if the argument contains the macro \endfoo, likewise that will spoof the result.

At the expense of expandability, I address Ulrich's concerns about spaces by calling on the facilities of tokcycle, which has no problem discerning spaces in its input stream. Thus, I show various examples and how much can be discerned about their content on the fly.

The only restriction is that the argument stream not contain the token endtokcycraw, which is reserved by the tokcycle package.

\documentclass{article}
\usepackage{tokcycle}
\newcommand\fooA{\ifx\foofirst\bgroup\expandafter\foodelim\else
\expandafter\fooundelim\fi}
\newcounter{tokcnt}
\newcommand\foodelim[1]{%
\setcounter{tokcnt}{0}%
\tokcycle
{\stepcounter{tokcnt}}%
{\stepcounter{tokcnt}}%
{#1}%
\tctestifnum{\value{tokcnt}=0}{\foonil}{%
\tctestifnum{\value{tokcnt}=1}{\fooone}{\foomulti}}{#1}}
\newcommand\fooundelim[1]{Undelimited argument to foo: #1}
\newcommand\foonil[1]{Delimited null argument to foo}
\newcommand\fooone[1]{Delimited single token to foo: #1%
\newcommand\foomulti[1]{Delimited multi-token to foo: #1%
\begin{document}
\foo a \par
\foo{} \par
\foo{ } \par
\foo{A} \par
\foo{ B} \par
\foo{C D} \par
\foo{ab}
\end{document}


• You can save \expandafters by rewriting \expandafter\ifx\expandafter\relax\detokenize{#2}\relax as \if\relax\detokenize{#2}\relax – Phelype Oleinik May 10 at 15:55
• @PhelypeOleinik Good point. – Steven B. Segletes May 10 at 15:56
• If I get this right, it strictly spoken is not a test for a single token but is a test for a single undelimited argument. It is relied on \foo's argument not being empty/blank and not containing the token \endfoo unless as 1st token. If \foo's argument has leading space-tokens these will be discarded instead of being considered when \fooaux's 1st undelimited argument is gathered. – Ulrich Diez May 11 at 12:40
• @UlrichDiez Please see my supplement – Steven B. Segletes May 11 at 14:56
• @StevenB.Segletes "Precision of terminology is good; yet as you note, it can be overwhelming to the newcomer." That's a true word, which I unfortunately do not always heed! ;-) Please feel free to remind me of it whenever you see the need of doing so. :-) From time to time I am sort of carried away when it comes to explaining concepts related to (La)TeX. Then my attempts to work out the context end up getting lost in subtle details, which doesn't really help people to get an overview first. – Ulrich Diez May 11 at 15:32

Preliminary remarks:

1. The special case of the argument being empty/consisting of zero tokens also is one of the cases where the argument is not a single token. (In the example below the corresponding expandable test is \UD@CheckWhetherNull.)

2. TeX strips off the outermost level of matching surrounding curly braces from a macro-argument if present: If \foo is a macro processing a single undelimited argument, then \foo x and \foo{x} yield the same.

3. Explicit space-tokens (explicit character-tokens of catcode 10 and charcode 32) need special treatment when processed via macros: When gathering tokens that form an undelimited argument TeX discards preceding explicit space-tokens. Therefore an explicit space-token can only be processed as undelimited macro-argument if nested in braces. (In the example below the corresponding expandable test is \UD@CheckWhetherLeadingExplicitSpace.)

4. Curly braces ({ or }) also get tokenized as tokens, as explicit character-tokens of catcode 1(begin group) or 2(end group): With \foo{{x}} the argument of \foo consists of the three tokens {1, x11 and }2. Thus this is a case where the argument is not a single token. (In the example below the corresponding expandable test is \UD@CheckWhetherBrace.)

In the example below with the expandable macro \CheckWhetherSingleToken the carrying out of the tests mentioned above is triggered via \romannumeral-expansion. Therefore in expansion-contexts the result is obtained after two expansion-steps, e.g., after "hitting" \CheckWhetherSingleToken via \expandafter twice/after "hitting" the first token of \CheckWhetherSingleToken's toplevel-expansion via \expandafter.

(The gist of \romannumeral-expansion is:

1. TeX expands expandable tokens while gathering the first token of a TeX-⟨number⟩-quantity that is to be converted by \romannumeral.
2. In case the value of the TeX-⟨number⟩-quantity gathered is not positive TeX will silently "eat"/discard the tokens forming the TeX-⟨number⟩-quantity without delivering anything in return.

Therefore you can (ab?)use \romannumeral for tricking TeX into doing a lot of expansion- and shuffling-around-of-tokens/macro-arguments-work as long as it is ensured that in the end the resulting token-sequence begins with a TeX-⟨number⟩-quantity whose value is not positive.)

\errorcontextlines=10000
\documentclass[a4paper]{article}

\makeatletter
%%=============================================================================
%% Paraphernalia:
%%    \UD@firstoftwo, \UD@secondoftwo, \UD@Exchange, \UD@PassFirstToSecond,
%%    \UD@removespace, \UD@stopromannumeral,  \UD@CheckWhetherNull,
%%=============================================================================
\newcommand\UD@firstoftwo[2]{#1}%
\newcommand\UD@secondoftwo[2]{#2}%
\newcommand\UD@Exchange[2]{#2#1}%
\newcommand\UD@PassFirstToSecond[2]{#2{#1}}%
\@ifdefinable\UD@removespace{\UD@Exchange{ }{\def\UD@removespace}{}}%
\@ifdefinable\UD@stopromannumeral{\chardef\UD@stopromannumeral=\^^00}%
%%-----------------------------------------------------------------------------
%% Check whether argument is empty:
%%.............................................................................
%% \UD@CheckWhetherNull{<Argument which is to be checked>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is empty>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is not empty>}%
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
\newcommand\UD@CheckWhetherNull[1]{%
\romannumeral\expandafter\UD@secondoftwo\string{\expandafter
\UD@secondoftwo\expandafter{\expandafter{\string#1}\expandafter
\UD@secondoftwo\string}\expandafter\UD@firstoftwo\expandafter{\expandafter
\UD@secondoftwo\string}\expandafter\UD@stopromannumeral\UD@secondoftwo}{%
\expandafter\UD@stopromannumeral\UD@firstoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether argument's first token is a catcode-1-character
%%.............................................................................
%% \UD@CheckWhetherBrace{<Argument which is to be checked>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked has a leading
%%                        explicit catcode-1-character-token>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked does not have a
\newcommand\UD@CheckWhetherBrace[1]{%
\romannumeral\expandafter\UD@secondoftwo\expandafter{\expandafter{%
\string#1.}\expandafter\UD@firstoftwo\expandafter{\expandafter
\UD@secondoftwo\string}\expandafter\UD@stopromannumeral\UD@firstoftwo}{%
\expandafter\UD@stopromannumeral\UD@secondoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether brace-balanced argument starts with a space-token
%%.............................................................................
%% \UD@CheckWhetherLeadingExplicitSpace{<Argument which is to be checked>}%
%%                                     {<Tokens to be delivered in case <argument
%%                                       which is to be checked> does have a
%%                                     {<Tokens to be delivered in case <argument
%%                                       which is to be checked> does not have a
\romannumeral\UD@CheckWhetherNull{#1}%
{\expandafter\UD@stopromannumeral\UD@secondoftwo}%
{%
% Let's nest things into \UD@firstoftwo{...}{} to make sure they are nested in braces
% and thus do not disturb when the test is carried out within \halign/\valign:
\expandafter\UD@firstoftwo\expandafter{%
\expandafter\expandafter\expandafter\UD@stopromannumeral
\romannumeral\expandafter\UD@secondoftwo
}{}%
}%
}%
\expandafter\UD@CheckWhetherNull\expandafter{\UD@firstoftwo{}#1}%
{\UD@Exchange{\UD@firstoftwo}}{\UD@Exchange{\UD@secondoftwo}}%
{\expandafter\expandafter\expandafter\UD@stopromannumeral
\expandafter\expandafter\expandafter}%
\expandafter\UD@secondoftwo\expandafter{\string}%
}%
}%
%%=============================================================================
\newcommand\CheckWhetherSingleToken[1]{%
\romannumeral\UD@CheckWhetherNull{#1}{\UD@secondoftwo}{%
\UD@CheckWhetherBrace{#1}{\UD@secondoftwo}{%
\expandafter\UD@CheckWhetherNull\expandafter{\UD@removespace#1}%
}{%
\expandafter\UD@CheckWhetherNull\expandafter{\UD@firstoftwo{}#1}%
}%
}%
}%
{\expandafter\UD@stopromannumeral\UD@firstoftwo}%
{\expandafter\UD@stopromannumeral\UD@secondoftwo}%
}%
%%=============================================================================
\newcommand\something[1]{This is something with single-token-argument: (#1)}
\newcommand\somethingelse[1]{This is something else with multi-token-argument or zero-token-argument=empty argument: (#1)}
\newcommand\foo[1]{%
\CheckWhetherSingleToken{#1}{\something}{\somethingelse}{#1}%
}%
\makeatother

\begin{document}

\noindent
\verb|\foo x| yields: \foo x\\
\verb|\foo{x}| yields: \foo{x}\\
\verb|\foo{{x}}| yields: \foo{{x}}\\
\verb|\foo{xy}| yields: \foo{xy}\\
\verb|\foo{{xy}}| yields: \foo{{xy}}\\
\verb|\foo{ }| yields: \foo{ }\\
\verb|\foo{ a}| yields: \foo{ a}\\
\verb|\foo{}| yields: \foo{}

\end{document}


• \foo{ä}` yields multi-token for pdfTeX (which is true or not, depending on what you are testing :) – Phelype Oleinik May 12 at 14:55
• @PhelypeOleinik The test is not about characters of .tex-input but is about tokens that came into being by tokenizing .tex-input. Of course the result depends both on the TeX-engine in use and on the encoding of the .tex-input-file in use. With .tex-input-files in utf-8-encoding/multibyte-encoding each character is formed by multiple bytes whereof each byte is taken for a(n active character-)token with traditional 8-bit TeX-engines like pdfLaTeX (if the inputenc-package with utf8-encoding is loaded, which is standard/the default nowadays). – Ulrich Diez May 21 at 0:30