# Problem with my understanding of expandafter and newcommand with arguments and sout

I've tried for days to get my head around the expansion of code. Sadly, I don't get it.

I try to pass a command with arguments as an argument to sout.

\documentclass{article}
\usepackage{ulem,lipsum}
\textwidth=3cm % just to force hyphenation/linebreaks to happen
\usepackage[T1]{fontenc}

\newcommand*{\rom}[1]{\romannumeral #1}

\newcommand\INTiintKATAkatNUMinum{% A bunch of commands get’s constructed by a lyx extension
Test Text for demonstration. In the original this command is constructed. %
}

\newcommand{\Kat}[4]{% Command to piece together the constructed commands.
\expandafter\csname INT\rom{#1}intKAT#2katNUM\rom{#3}num#4\endcsname%
}

\newcommand\soutt[1]{% sout with expansion
\expandafter\sout\expandafter{#1}%
}

\begin{document}
\sout{The sout makro of ulem is able to handle hyphenation and linebreaks in text entered directly but not as macro see next line!}
\soutt{\INTiintKATAkatNUMinum} %works
\soutt{\Kat{1}{A}{1}{}} %doesn't
\end{document}

Any help is very much appreciated.

• BTW, you can use \romannumeral #1\relax to make sure no additional digits are appended. – John Kormylo Feb 6 at 15:52

You should be aware of the fact that \Kat requires more than one expansion step in order to deliver the stored text: after the first expansion step you get

\expandafter\csname INT\rom{1}intKATAkatNUM\rom{1}num\endcsname

This \expandafter does nothing at all, because it tries to expand I.

You can force “almost full” expansion, which is what you need here, with the \romannumeral trick:

\documentclass{article}
\usepackage{ulem,lipsum}
\usepackage[T1]{fontenc}

\newcommand*{\rom}[1]{\romannumeral #1}

\newcommand\INTiintKATAkatNUMinum{%
Test Text for demonstration. In the original this command is constructed. %
}

\newcommand{\Kat}[4]{% Command to piece together the constructed commands.
\csname INT\rom{#1}intKAT#2katNUM\rom{#3}num#4\endcsname
}

\newcommand\soutt[1]{% sout with expansion
\expandafter\sout\expandafter{\romannumeral-Q#1}%
}

\begin{document}

\parbox{2cm}{
\sout{The sout makro of ulem is able to handle hyphenation and linebreaks in text entered directly but not as macro see next line!}
\soutt{\INTiintKATAkatNUMinum} %works
\soutt{\Kat{1}{A}{1}{}} %doesn't
}

\end{document}

And no, \sout doesn't do hyphenation.

The \romannumeral trick has been explained several times, but here's a brief illustration.

The primitive \romannumeral wants a <number> after it and here we exploit a peculiar feature of TeX: after an explicit <number>, TeX looks for an <optional space> after it, expanding tokens during this lookup; it stops expansion upon finding either an explicit space token or an unexpandable token.

An “explicit <number>” can be expressed in several ways:

• a sequence of digits;
• a hexadecimal number, that is, a sequence of digits or characters among ABCDEF, if the first token is ";
• an octal number, that is, a sequence of digits among 01234567 if the first token is ';
• an alphabetical constant, if the first token is  (a backquote).

In all cases, a minus sign should precede the radix notation "', if we want to specify a negative number.

In the first three cases, TeX performs expansion until finding something that doesn't qualify as an admissible digit.

The case that interests us is the last one. An alphabetic constant is a character token or a control sequence whose name consists of a single character (and need not be defined). So

Q  \Q

are equivalent ways to ask for number 81 (the ASCII code of Q). The escaped notation is essential for characters with “strange” category code, such as

\% \^^@

(the latter is 0).

The fact that TeX expands tokens after the alphabetical constants is decisive for this application: the <number> is already fully known, and in our case it is -81, but TeX expands tokens nonetheless. After ending the search for the optional space, TeX will proceed to finalize the expansion of \romannumeral-Q, which is empty because the number is negative.

We cannot use \romannumeral-81 directly, because if the following token after expansion happens to be a digit, it would be taken as part of the <number>.

A limitation of this method is that an initial space in the final expansion will be gobbled.

In the next release of TeX Live, all engines will have \expanded (currently only available with LuaTeX); MiKTeX based engines should already have it. With \expanded, the thing will be simpler:

\newcommand\soutt[1]{% sout with expansion
\expandafter\sout\expandafter{\expanded{#1}}
}

because \expanded delivers the full expansion of its argument in a single step.

You can use soul, if you want hyphenation. The corresponding macro is \st (or \textst). Just for a change, I'll implement it in expl3, where the \romannumeral is available as “f-expansion”.

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{soul}
\usepackage{xparse}

\ExplSyntaxOn

\NewDocumentCommand{\Kat}{mmmm}
{
\use:c
{
INT \int_to_roman:n {#1} intKAT #2 katNUM \int_to_roman:n {#3} num #4
}
}
\NewExpandableDocumentCommand{\soutt}{m}
{
\oberreiter_sout:f { #1 }
}
\cs_new_protected:Nn \oberreiter_sout:n { \textst{#1} }
\cs_generate_variant:Nn \oberreiter_sout:n { f }

\ExplSyntaxOff

\newcommand\INTiintKATAkatNUMinum{%
Test Text for demonstration. In the original this command is constructed. %
}

\begin{document}

\parbox{2cm}{
\st{The st macro of soul is able to handle hyphenation and linebreaks
in text entered directly but not as macro see next line!}
\soutt{\INTiintKATAkatNUMinum} %works
\soutt{\Kat{1}{A}{1}{}} %doesn't
}

\end{document}

• I always upvote the roman numeral trick, even though I still don't fully understand it. But I am presently looking at texdev.net/2011/07/05/expansion-using-romannumeral – Steven B. Segletes Feb 6 at 14:18
• @StevenB.Segletes I added some explanations and fixed the copy-paste error. – egreg Feb 6 at 14:41
• Thank you for the concise explanation. Does \expanded work like \romannumeral (expanding to the first non-expandable token) or does it work by replacing its complete argument with the \edef'ed equivalent? – Steven B. Segletes Feb 6 at 14:54
• @StevenB.Segletes It does the same as \edef: it is the expandable version of \begingroup\edef\x{\endgroup<tokens>}\x – egreg Feb 6 at 15:04
• After searching as long as I did for a solution. I thought it would be one of those long elaborated edev things. I’m always surprised how you wizards come up with such elegant solutions. Thx a lot! – Andreas Oberreiter Feb 6 at 18:48

Inspired by egreg's answer—where he exhibits the \romannumeral-Q⟨argument⟩-trick for keeping expansion with the leading token of the ⟨argument⟩ going until obtaining a leading token which is not expandable, whereby \romannumeral-expansion might "eat" a leading space-token from the argument—I just wrote a routine \HitArgsFirstTokenWithExpandafters which does "hit" the leading token of its undelimited/curly-brace-nested argument with \expandafter until obtaining a leading token which is not expandable.
Even if the argument was a single token, not nested in curly braces, the result will be delivered nested in curly braces.

The gist of the routine is:

A loop is initiated:

The routine checks whether the argument is empty or whether the argument's leading token is a space or an opening brace.
If this is the case, the routine is done as emptiness or leading braces or leading spaces imply that there is no expandable leading token.

If this is not the case, one can easily extract the leading token from the argument and check expandability of that token via \expandafter\ifx\noexpand⟨token⟩⟨token⟩⟨not expandable⟩\else ⟨expandable⟩\fi and in case of expandability produce a hit via \expandafter before initiating the next iteration.

Be aware that this routine does not expand the argument totally. Like the \romannumeral-Q⟨argument⟩-trick it aims only at the very first token of the argument (but unlike the \romannumeral-Q⟨argument⟩-trick it will not "eat" a leading space-token).
Thus breaking of lines will still be broken in case the argument contains macro tokens but not as leading tokens but somewhere behind the (fully expanded) leading tokens.

Be aware that such routines/tricks might cause problems, e.g., with unbalanced braces in case there are leading expandable token-sequences like
\expandafter\@gobble\string{!!!Attention!!! the closing brace will be unbalanced}.

Be aware that such routines/tricks might cause problems (capacity exceeded errors, never terminating loops and the like), e.g., with macros that expand to themselves, e.g., things like \foo after defining \newcommand\foo{\foo⟨something or nothing⟩}.

Other approaches aiming at expanding the argument totally and already mentioned in egreg's answer are using \expanded with LuaTeX-based engines and applying some variant of \edef—I suggest \protected@edef—with engines that don't have the \expanded-primitive.

\documentclass{article}
\usepackage{ulem,lipsum}
%\usepackage[T1]{fontenc}
\usepackage{ifluatex}

\newcommand*{\rom}[1]{\romannumeral #1}

\newcommand\INTiintKATAkatNUMinum{ A bunch of commands gets constructed by a lyx extension
Test Text for demonstration. In the original this command is constructed. %
}

\newcommand{\Kat}[4]{% Command to piece together the constructed commands.
\expandafter\csname INT\rom{#1}intKAT#2katNUM\rom{#3}num#4\endcsname
}%

% a redefinition of \sout which lets you see what tokens \sout gets as argument:
%\def\sout#1{\def\test{#1}\texttt{\meaning\test}}%

\newcommand\soutt[1]{% expandable sout with total expansion of first token of argument
\expandafter\expandafter\expandafter\sout\HitArgsFirstTokenWithExpandafters{#1}%
}%

% You can use \protected@edef for defining a temporary macro to expand
% to the total expansion of the argument before passing the expansion of
% that temporary macro to \sout:

\newcommand\souttb[1]{% non-expandable sout with total expansion of entire argument
\csname protected@edef\endcsname\mytempa{#1}%
\expandafter\sout\expandafter{\mytempa}%
}%

% In case of using LuaLaTeX you can use the \expanded-primitive:

\ifluatex
\newcommand\souttc[1]{% expandable sout with total expansion of entire argument
\expandafter\sout\expandafter{\expanded{#1}}%
}%
\fi

\makeatletter
%%-----------------------------------------------------------------------------
%% Paraphernalia ;-) :
%%.............................................................................
\newcommand\UD@firstoftwo[2]{#1}%
\newcommand\UD@secondoftwo[2]{#2}%
\newcommand\UD@exchange[2]{#2#1}%
%%-----------------------------------------------------------------------------
%% Check whether argument is empty:
%%.............................................................................
%% \UD@CheckWhetherNull{<Argument which is to be checked>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is empty>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is not empty>}%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
%%.............................................................................
\newcommand\UD@CheckWhetherNull[1]{%
\romannumeral0\expandafter\UD@secondoftwo\string{\expandafter
\UD@secondoftwo\expandafter{\expandafter{\string#1}\expandafter
\UD@secondoftwo\string}\expandafter\UD@firstoftwo\expandafter{\expandafter
\UD@secondoftwo\string}\expandafter\expandafter\UD@firstoftwo{ }{}%
\UD@secondoftwo}{\expandafter\expandafter\UD@firstoftwo{ }{}\UD@firstoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether argument's first token is a catcode-1-character
%%.............................................................................
%% \UD@CheckWhetherBrace{<Argument which is to be checked>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked has leading
%%                        catcode-1-token>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked has no leading
%%                        catcode-1-token>}%
\newcommand\UD@CheckWhetherBrace[1]{%
\romannumeral0\expandafter\UD@secondoftwo\expandafter{\expandafter{%
\string#1.}\expandafter\UD@firstoftwo\expandafter{\expandafter
\UD@secondoftwo\string}\expandafter\expandafter\UD@firstoftwo{ }{}%
\UD@firstoftwo}{\expandafter\expandafter\UD@firstoftwo{ }{}\UD@secondoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether brace-balanced argument starts with a space-token
%%.............................................................................
%% \UD@CheckWhetherLeadingSpace{<Argument which is to be checked>}%
%%                             {<Tokens to be delivered in case <argument
%%                               which is to be checked>'s 1st token is a
%%                               space-token>}%
%%                             {<Tokens to be delivered in case <argument
%%                               which is to be checked>'s 1st token is not
%%                               a space-token>}%
\romannumeral0\UD@CheckWhetherNull{#1}%
{\expandafter\expandafter\UD@firstoftwo{ }{}\UD@secondoftwo}%
}%
\expandafter\UD@CheckWhetherNull\expandafter{\UD@secondoftwo#1{}}%
{\UD@exchange{\UD@firstoftwo}}{\UD@exchange{\UD@secondoftwo}}%
{\UD@exchange{ }{\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter}\expandafter\expandafter
\expandafter}\expandafter\UD@secondoftwo\expandafter{\string}%
}%
%%-----------------------------------------------------------------------------
%% Extract first inner undelimited argument:
%%.............................................................................
%%   \UD@ExtractFirstArg{ABCDE} yields  {A}
%%
%%   \UD@ExtractFirstArg{{AB}CDE} yields  {AB}
%%.............................................................................
\newcommand\UD@RemoveTillUD@SelDOm{}%
\long\def\UD@RemoveTillUD@SelDOm#1#2\UD@SelDOm{{#1}}%
\newcommand\UD@ExtractFirstArg[1]{%
\romannumeral0%
\UD@ExtractFirstArgLoop{#1\UD@SelDOm}%
}%
\newcommand\UD@ExtractFirstArgLoop[1]{%
\expandafter\UD@CheckWhetherNull\expandafter{\UD@firstoftwo{}#1}%
{ #1}%
{\expandafter\UD@ExtractFirstArgLoop\expandafter{\UD@RemoveTillUD@SelDOm#1}}%
}%
%%-----------------------------------------------------------------------------
%% Check whether argument's first token is expandable
%%.............................................................................
%% \UD@CheckWhetherFirstTokenExpandable{<Argument which is to be checked>}%
%%                      {<Tokens to be delivered in case that
%%                         <argument which is to be checked> has a first
%%                         token which is expandable>}%
%%                      {<Tokens to be delivered in case that
%%                         <argument which is to be checked> does not have
%%                         a first token which is expandable>}%
%%
\newcommand\UD@CheckWhetherFirstTokenExpandable[1]{%
\romannumeral0%
\UD@CheckWhetherNull{#1}{\UD@exchange{ }{\expandafter}\UD@secondoftwo}{%
\UD@CheckWhetherBrace{#1}{\UD@exchange{ }{\expandafter}\UD@secondoftwo}{%
\expandafter\expandafter\expandafter\UD@@CheckWhetherFirstTokenExpandable
\UD@ExtractFirstArg{#1}%
}%
}%
}%
}%
\newcommand\UD@@CheckWhetherFirstTokenExpandable[1]{%
\expandafter\ifx\noexpand#1#1%
\expandafter\UD@firstoftwo\else\expandafter\UD@secondoftwo\fi
{\UD@exchange{ }{\expandafter}\UD@secondoftwo}%
{\UD@exchange{ }{\expandafter}\UD@firstoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Hit argument's first token with \expandafter until argument's first token is
%% not an expandable token. Then deliver that nested in braces.
%%.............................................................................
\newcommand\HitArgsFirstTokenWithExpandafters[1]{%
\romannumeral
\UD@CheckWhetherFirstTokenExpandable{#1}{%
\expandafter\UD@firstoftwo\expandafter{\expandafter}%
\romannumeral0\UD@exchange{ }{\expandafter\expandafter\expandafter}%
\expandafter\HitArgsFirstTokenWithExpandafters\expandafter{#1}%
}{0 {#1}}%
}%
%%
%% With all the macros above, the result is delivered after two expansion-steps
%% / after two \expandafter-chains due to  \romannumeral-expansion.
\makeatother

\parindent=0ex
\parskip=\smallskipamount

\begin{document}
\vspace*{-4.5cm}%
\enlargethispage{8cm}%
\pagestyle{empty}%
The sout makro of ulem does suppress hyphenation in any case.

\hrulefill\null

\begin{verbatim}
\parbox{4.5cm}{%
\sout{The sout makro of ulem is able to handle linebreaks in text
entered directly but not as macro!}
}%
\end{verbatim}%
\parbox{5cm}{%
\sout{The sout makro of ulem is able to handle linebreaks in text
entered directly but not as macro!}
}%

\hrulefill\null

Linebreaking is not disturbed with the following:

\verb|\parbox{5cm}{\soutt{\INTiintKATAkatNUMinum}}|

\parbox{5cm}{\soutt{\INTiintKATAkatNUMinum}}%

\dotfill\null

\verb|\parbox{5cm}{\soutt{\Kat{1}{A}{1}{}}}|

\parbox{5cm}{\soutt{\Kat{1}{A}{1}{}}}

\hrulefill\null

Linebreaking is disturbed with the following:

\verb|\parbox{5cm}{\soutt{Breaking of lines is broken: \INTiintKATAkatNUMinum}}|

\parbox{5cm}{\soutt{Breaking of lines is broken: \INTiintKATAkatNUMinum}}

\dotfill\null

\verb|\parbox{5cm}{\soutt{Breaking of lines is broken: \Kat{1}{A}{1}{}}}|

\parbox{5cm}{\soutt{Breaking of lines is broken: \Kat{1}{A}{1}{}}}

\hrulefill\null

Linebreaking is not disturbed with the following:

\verb|\parbox{5cm}{\souttb{Breaking of lines is not broken: \INTiintKATAkatNUMinum}}|

\parbox{5cm}{\souttb{Breaking of lines is not broken: \INTiintKATAkatNUMinum}}

\dotfill\null

\verb|\parbox{5cm}{\souttb{Breaking of lines is not broken: \Kat{1}{A}{1}{}}}|

\parbox{5cm}{\souttb{Breaking of lines is not broken: \Kat{1}{A}{1}{}}}

\ifluatex\dotfill\null

\verb|\parbox{5cm}{\souttc{Breaking of lines is not broken: \INTiintKATAkatNUMinum}}|

\parbox{5cm}{\souttc{Breaking of lines is not broken: \INTiintKATAkatNUMinum}}

\dotfill\null

\verb|\parbox{5cm}{\souttc{Breaking of lines is not broken: \Kat{1}{A}{1}{}}}|

\parbox{5cm}{\souttc{Breaking of lines is not broken: \Kat{1}{A}{1}{}}}
\fi

\end{document}