# Why parameter whose tokens are being iterated through should be ended with a non-empty macro, and what should that macro actually contain?

I've asked this question more than once here, but all I could get is that "it's safer" to use a non-empty macro. I was never told why, and (more importantly) it was never clarified what exactly the macro should contain.

This code sample is a "looping" macro that iterates trough each token of its parameter:

\def\zEnd{\zEnd}
\def\zzIterator#1{%
\ifx#1\zEnd
\else
#1%
\expandafter\zzIterator
\fi
}
\def\zIterator#1{\zzIterator#1\zEnd}
\zIterator{hello}


There are 2 questions:

1. why \def\zEnd{\zEnd} should not be empty (should not be \def\zEnd{})?
2. does it really matter what exact macro is inside the curly braces of \zEnd, whether the enclosed macro is itself \zEnd or anything else (like \zIterator, \zzIterator, or whatnot)?

The reason for using a 'quark' (a macro defined to expand to itself) is that it can never match anything else. If you use

\def\zEnd{}


then your marker is \ifx equal to \empty, so something like

\zIterator{hello\empty more tokens}


will stop early. It's the same with any other definition other than the marker itself: you need to be sure that the token is not in the input. Using a quark, we only have one 'restricted' token.

This code sample is a "looping" macro that iterates trough each token of its parameter:

\def\zEnd{\zEnd}
\def\zzIterator#1{%
\ifx#1\zEnd
\else
#1%
\expandafter\zzIterator
\fi
}
\def\zIterator#1{\zzIterator#1\zEnd}
\zIterator{hello}


There are 2 questions:

1. why \def\zEnd{\zEnd} should not be empty (should not be \def\zEnd{})?
2. does it really matter what exact macro is inside the curly braces of \zEnd, whether the enclosed macro is itself \zEnd or anything else (like \zIterator, \zzIterator, or whatnot)?

The token \zEnd is used as "sentinel token" for ending iteration/recursion.

Ad 1: The \ifx-comparison triggers termination of the loop by comparing meanings of tokens. In practise there won't be many tokens besides \zEnd which are defined to expand to \zEnd. But there may be many short macros that at some stage (temporarily) are defined to not deliver any token at all. Such macro-tokens would erroneously terminate the loop if \zend was defined as \def\zEnd{}.

Ad 2: It doesn't matter. What does matter is that the sentinel-token is choosen/defined in a way where it can be taken for granted/ensured that tokens of equal meaning do not occur within the argument/within the set of tokens that is to be iterated.

By the way: The statement "iterates trough each token of its parameter" is wrong:

The underlying looping macro \zzIterator does not process a single token but does proces a non-delimited argument. Therefore the looping macro does not iterate through each token of its parameter but iterates its parameter non-delimited-argument-wise until either encountering a non-delimited argument which has two leading tokens of equal meaning or encountering a token whose meaning equals the meaning of \zEnd.

"Non-delimited-argument-wise" implies the removal of one level of curly braces if present and the discarding of explicit space-tokens.

See, e.g., what happens with:

\def\zEnd{\zEnd}
\def\zzIterator#1{%
\ifx#1\zEnd
Iteration ends.
\else
Iteration processes: #1 %
\expandafter\zzIterator
\fi
}
\def\zIterator#1{\zzIterator#1\zEnd}
\def\gobble#1{}%
\let\foobar=\zEnd

\zIterator{hel\foobar lo\gobble}
\bigskip

\zIterator{h{ell}o}
\bigskip

\zIterator{ {h} {ell} {o} }
\bigskip

\zIterator{he{ll\gobble}o\gobble}
\bigskip

% Don't do:
%\zIterator{he{ll}o}
% Here \ifx will compare l to l and then \zEnd will be carried out,
% yielding a never terminating recursion as \zEnd calls itself again
% and again...
\bye


Reversing the tokens to be compared by \ifx is a bit safer as you won't get into the \zIterator{he{ll}o}-pitfall:

\def\zEnd{\zEnd}
\def\zzIterator#1{%
\ifx\zEnd#1%<---- reversed.
\else
#1%
\expandafter\zzIterator
\fi
}
\def\zIterator#1{\zzIterator#1\zEnd}
\zIterator{hello}


The loop can still be outmanoeuvred by tokens equal to \zEnd and by (unbalanced) \if../\else/\fi in the argument.

Detecting emptiness of arguments is a bit safer:

\long\def\firstoftwo#1#2{#1}
\long\def\secondoftwo#1#2{#2}
%%=============================================================================
%% Check whether argument is empty:
%%=============================================================================
%% \CheckWhetherNull{<Argument which is to be checked>}%
%%                  {<Tokens to be delivered in case that argument
%%                    which is to be checked is empty>}%
%%                  {<Tokens to be delivered in case that argument
%%                    which is to be checked is not empty>}%
%%
%% Due to \romannumeral0-expansion the result is delivered after two
%% expansion-steps/after two "hits" by \expandafter.
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
%%
\long\def\CheckWhetherNull#1{%
\romannumeral0\expandafter\secondoftwo\string{\expandafter
\secondoftwo\expandafter{\expandafter{\string#1}\expandafter
\secondoftwo\string}\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}\firstoftwo\expandafter{} \secondoftwo}%
{\firstoftwo\expandafter{} \firstoftwo}%
}%
%%=============================================================================
\long\def\IIterator#1#2{%
\expandafter\CheckWhetherNull\expandafter{\firstoftwo{}#1}{%
Iteration terminated.%
}{%
Processing: #2 %
\expandafter\IIterator\expandafter{\firstoftwo{}#1}%
}%
}%
\long\def\Iterator#1{\IIterator{#1{}}#1{}}%
%%
\Iterator{hello}%
\bigskip\par
\Iterator{he{ll}o}%
\bigskip\par
Of course this does---like the other variants---not take space-tokens
into account because \firstoftwo{\TeX}{} discards space-tokens that
precede non-delimited arguments:
\bigskip\par
\Iterator{ h e l l o }%

\bye