3

(This question is not about bugs or errors or whatsoever misbehavior. I just would like to know how a pleasant trick was done.)

I just encountered another subtlety in which the way of xparse's handling of things differs from the old ways:

If defining via \newcommand a macro which processes an optional argument, then the mechanism \protected@testopt\@testopt\kernel@ifnextchar for looking for an optional argument in any case removes spaces while "looking" for the presence of an optional argument. Spaces are removed also when no optional argument is present.
(This is not necessarily a mis-conception: With macros defined in terms of \newcommand the optional argument usually comes directly behind the control-word-token that forms that macro. Thus, if that control-word-token comes from reading and tokenizing tex-input, then TeX's reading-apparatus is in state S(skipping blanks) after tokenizing it. Therefore in this case subsequent spaces of the tex-input are skipped so that no space-tokens would come from them which the \protected@testopt\@testopt\kernel@ifnextchar could "see". The mechanism's space-removal also when no optional argument is present is only relevant in situations where the optional argument is searched right after tokenizing some token after whose tokenization the reading-apparatus is in state m (middle of line).)

Seems xparse's routine for looking for an optional argument, which is used with macros defined in terms of \NewDocumentCommand, removes spaces only in case an optional argument is present.
Seems xparse's routine for looking for an optional argument, which is used with macros defined in terms of \NewDocumentCommand, preserves spaces in case an optional argument is not present.

My question is:

Can you outline how xparse does the trick of preserving spaces in case an optional argument is not present?

(Checking the meaning of the next token via \ifx after "peeking ahead" via \futurelet (or via \afterassignment\checknexttokensmeaning\let\nexttoken= ) for detecting a space might be part of the trick, but the space might not be an explicit space-token but an implicit space, e.g., a control-sequence-token let equal to the space-token or an active space let equal to the space-token. Checking meanings alone could not crank out these cases.)

In case you need to see it, here is a small example exhibiting the subtle difference—please don't be confused by \exchange, \expandafter and \csname...\endcsname—I just did some trickery in order to get the -NoValue--marker into the default-value of the optional argument within the definition done in terms of \newcommand, so that \IfNoValueF can be used within the \newcommand-definition, too:

\documentclass{article}
%\usepackage{xparse}
%
\NewDocumentCommand\testdocumentcommand{mo}{\{#1\}\IfNoValueF{#2}{[#2]}}%
%
\newcommand\testnewcommand[1]{\{#1\}\testnewcommandoptarg}%
\begingroup\def\exchange#1#2{#2#1}
\expandafter\expandafter\expandafter\exchange
\expandafter\expandafter\expandafter{\csname c_novalue_tl\endcsname}%
{\endgroup\newcommand\testnewcommandoptarg[1][}]{\IfNoValueF{#1}{[#1]}}%

\begin{document}

\noindent Here the results differ:

\noindent With \verb|\testdocumentcommand|'s routine for looking for
optional arguments the space between \verb|{A}| and \verb|C| is not removed in case 
of no optional argument being present.

\noindent With \verb|\testnewcommand|'s routine for looking for
optional arguments the space between \verb|{A}| and \verb|C|  is removed in case 
of no optional argument being present.

\medskip

\verb*|\testdocumentcommand{A} C|: \testdocumentcommand{A} C

\verb*|\testnewcommand{A} C|: \testnewcommand{A} C

\medskip\hrule\medskip

\noindent Here the results are the same as no space is present between \verb|{A}| and \verb|C|:

\verb*|\testdocumentcommand{A}C|: \testdocumentcommand{A}C

\verb*|\testnewcommand{A}C|: \testnewcommand{A}C

\medskip\hrule\medskip

\noindent Here the results are the same as the optional argument \verb|[B]| is present:

\verb*|\testdocumentcommand{A}  [B]C|: \testdocumentcommand{A}  [B]C

\verb*|\testnewcommand{A}  [B]C|: \testnewcommand{A}  [B]C

\medskip\hrule\medskip

\noindent Here the results are the same as the optional argument \verb|[B]| is present:

\verb*|\testdocumentcommand{A}  [B] C|: \testdocumentcommand{A}  [B] C

\verb*|\testnewcommand{A}  [B] C|: \testnewcommand{A}  [B] C

\end{document}

enter image description here

1 Answer 1

3

I just found in xparse.dtx that with unexpandable commands internally \@@_peek_nonspace:NTF is used for preserving spaces while searching for optional arguments.

\@@_peek_nonspace:NTF in turn is defined in ltcmd.dtx:

% \begin{macro}{\@@_peek_nonspace:NTF, \@@_peek_nonspace_remove:NTF, \@@_peek_nonspace_aux:nNNTF}
%   Collect spaces in a loop, and put the collected spaces back in the
%   false branch of a call to \cs{peek_meaning:NTF} or
%   \cs{peek_meaning_remove:NTF}.
%    \begin{macrocode}
\cs_new_protected:Npn \@@_peek_nonspace:NTF
  { \@@_peek_nonspace_aux:nNNTF { } \@@_peek_meaning:NTF }
\cs_new_protected:Npn \@@_peek_nonspace_remove:NTF
  { \@@_peek_nonspace_aux:nNNTF { } \@@_peek_meaning_remove:NTF }
\cs_new_protected:Npn \@@_peek_nonspace_aux:nNNTF #1#2#3#4#5
  {
    \peek_meaning_remove:NTF \c_space_token
      { \@@_peek_nonspace_aux:nNNTF { #1 ~ } #2 #3 {#4} {#5} }
      { #2 #3 { #4 } { #5 #1 } }
  }
%    \end{macrocode}

interface3.pdf in turn says about \peek_meaning_remove:NTF:

\peek_meaning_remove:NTF ⟨test token⟩ {⟨true code⟩} {⟨false code⟩}

Tests if the next ⟨token⟩ in the input stream has the same meaning as the ⟨test token⟩ (as defined by the test \token_if_eq_meaning:NNTF). Spaces are respected by the test and the ⟨token⟩ is removed from the input stream if the test is true. The function then places either the ⟨true code⟩ or ⟨false code⟩ in the input stream (as appropriate to the result of the test).

Strictly spoken \@@_peek_nonspace:NTF does not collect explicit space-tokens from the token-stream but does via \peek_meaning_remove:NTF remove tokens from the token-stream whose meanings equal the meaning of the space-token and accumulate as many explicit space tokens as explicit/implicit space-tokens were removed. Thus with unexpandable commands the case of there being implicit space tokens is not cranked out but implicit space-tokens are replaced by explicit ones.

With expandable commands the situation of needing to preserve spaces in case of an optional argument not being present cannot occur as with expandable commands the last argument being an optional one is forbidden.

But due to the optional-argument-searching-routine for non-expandable commands replacing implicit space-tokens by explicit space-tokens, in case of the optional argument being between mandatory arguments there is a subtle difference with the behavior of unexpandable versus expandable commands regarding searching optional arguments and grabbing subsequent non-optional arguments:

\documentclass{article}
\usepackage{xparse}

\NewDocumentCommand\Unexptest{mom}{\{#1\}\IfNoValueF{#2}{[#2]}\{#3\}}

\NewExpandableDocumentCommand\exptest{mom}{\{#1\}\IfNoValueF{#2}{[#2]}\{#3\}}

\begin{document}

\makeatletter

The unexpandable variant does not take \verb|\@sptoken| for the second mandatory argument: The routine for searching the optional argument replaces them by explicit space tokens which (unlike implicit space tokens) in turn get discarded when \TeX\csname @sptoken\endcsname gathers the second mandatory argument.

The expandable variant takes the first \verb|\@sptoken| for the second mandatory argument because here with the routine for sarching the optional argument replacement of implicit spaces with explicit spaces does not occur.

\verb|\Unexptest{A}\@sptoken\@sptoken{C}|: \Unexptest{A}\@sptoken\@sptoken{C}

\verb|\exptest{A}\@sptoken\@sptoken{C}|: \exptest{A}\@sptoken\@sptoken{C}

\medskip\hrule\medskip

\verb|\Unexptest{A} {C}|: \Unexptest{A} {C}

\verb|\exptest{A} {C}|: \exptest{A} {C}

\end{document}

enter image description here

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .