Parsing a mix of plain text and commands

Question

I am trying to implement a command that parses some expression that can be a mix of plain text, commands with their (any number of) arguments, or even -- if I understood that correctly -- unexpandable parts.

To the end of parsing, I wish to scan the input 'item' by 'item', where an item is either a single char, maybe a {} pair, any command along with its arguments, or a single unexpandable stuff.

My first attempt was to recursively scan the input with :

\NewDocumentCommand\scan{m u{\relax}}{%
    #1%
    \ifblank{#2}
        {}{,\scan#2\relax}%
}

that renders a,b,c,d for \scan abcd\relax. However, commands must be wrapped for them to work, as in : \scan a{\vec x}bcd\relax, which correctly renders a,x,b,c,d (where x has a vector arrow). Depending on how they are defined, even nullary commands need wrapping. This requirement is very annoying to me so I wish to get rid of it. Furthermore, a wrapped {⋅⋅⋅} would eventually need recursive parsing so protecting commands with it is ultimately not acceptable since it wouldn't fix the issue anymore.

My second attempt was to use xstring on the input, unfortunately it only works with plain text.

My third attempt was to detokenize/retokenize. But I can't find how to make \tokenize work, and didn't find documentation on it : \tokenize{\detokenize{⋅⋅⋅}} for instance yields unrelated errors. Also I'm not convinced this would lead to a practical solution since that would leave me with the problem of grabbing commands with all their arguments, not mentionning the potential trouble of spaces removal.

Finally, I went back to the first attempt but with \expandafter in the hope of transforming a command and its arguments into a unique 'solid' thing, as in :

\NewDocumentCommand\scan{m u{\relax}}{%
    #1%
    \ifblank{#2}
        {}{,\expandafter\scan#2\relax}%
}
\NewDocumentCommand\scanit{m}{%
    \expandafter\scan#1\relax%
}

Which I can't get to compile when input contains commands, for instance I would like \(scanit {\vec abcd}\) to render a,b,c,d (where a has a vector arrow), but I get : Extra }, or forgotten $. \(\scanit {\vec abcd}\). In fact, there are cases where it doesn't work but it would without \expandafter -- e.g. with some nullaries.

Obviously I'm not in control of everything i'm doing here, so any explanation is appreciated

Answer 1

It is not exactly clear what you are trying to achieve, but the tokcycle package is set up to go through each token of an input stream and process them according to user defined directives. Tokens and braced material are categorized as Characters, Groups, Macros, or Spaces (CGMS), each of which gets it own directive.

While it can be told to stream the output directly as the input is processed, this only works for simple character input streams. If macros are involved, you can't execute the macro until its arguments have been processed. Thus, the typical and recommended mode of operation is to process the input and collect the processed tokens in a provided token list called \cytoks. The calls to \addcytoks are calls to add processed output to the token list. Then, once the end of the environment is reached, the processed \cytoks token list is rendered.

The #1 in the directives refers to the "current token" being processed. Throughout the environment, all tokens of a given type C, G, M, or S will successively go through their same respective directive and be processed according to the user defined directions. The default directives merely echo the input unchanged to the \cytoks list.

See the documentation and examples document at https://ctan.org/pkg/tokcycle

EDITED to handle regular braced-argument syntax.

\documentclass{article}
\usepackage{tokcycle}
\newif\ifvecnext
\stripgroupingtrue
\Characterdirective{\tctestifcon\ifvecnext
  {\addcytoks{\ensuremath{\vec{#1}}}}%
  {\addcytoks{#1}}%
  \vecnextfalse
}
\Groupdirective{\tctestifcon\ifvecnext
  {\addcytoks{\ensuremath{\vec{#1}}}}%
  {\groupedcytoks{\processtoks{#1}}}%
  \vecnextfalse
}
\Macrodirective{\tctestifx{\vec#1}
  {\vecnexttrue}%
  {\addcytoks{#1}}%
}\begin{document}
\tokencyclexpress \vec a,b,\vec{cx},d\endtokencyclexpress
\end{document}

To create your own named tokcycle environment, there is this:

\documentclass{article}
\usepackage{tokcycle}
\newif\ifvecnext
\stripgroupingtrue
\tokcycleenvironment\myscan% NEW TOKCYCLE ENVIRONMENT
{\tctestifcon\ifvecnext
  {\addcytoks{\ensuremath{\vec{##1}}}}%
  {\addcytoks{##1}}%
  \vecnextfalse
}% ↑ CHARACTER DIRECTIVE
{\tctestifcon\ifvecnext
  {\addcytoks{\ensuremath{\vec{##1}}}}%
  {\groupedcytoks{\processtoks{##1}}}%
  \vecnextfalse
}% ↑ GROUP DIRECTIVE
{\tctestifx{\vec##1}
  {\vecnexttrue}%
  {\addcytoks{##1}}%
}% ↑ MACRO DIRECTIVE
{\addcytoks{##1}}% ←DEFAULT SPACE DIRECTIVE
\begin{document}
\myscan \vec a,b,\vec{cx},d\endmyscan
\end{document}

---

SUPPLEMENT

As described in the SUPPLEMENT to this answer, LaTeX3: Elegant way to forward-reference a counter with a "future" value, I am working on package improvements to allow one to have the tokcycle directives access the future input stream directly. This will allow one to avoid the process of setting a flag in the Macro directive that is monitored by the Character directive. (EDIT: tokcycle[2021-05-27] has been released, so \makeatletter code has been removed from this MWE)

I envision the ability to pop and push tokens from the input stream. You can see that I avoid the Character directive altogether. In the \apply macro, I make use of two interesting package macros, \groupedcytoks and \tcpopliteral. Read all about them in the package documentation at https://ctan.org/pkg/tokcycle. Here, \groupedcytoks is used to introduce an additional group level into the output stream, namely, the argument to \ensuremath. The macro \tcpopliteral is used to pop an argument from the input stream, while preserving any leading spaces and grouping of the popped tokens (unlike normal TeX absorption that strips leading space as well as cat-1,2 tokens from the absorption of a group).

\documentclass{article}
\usepackage{tokcycle}
\Macrodirective{\ifx\vec#1\apply{#1}\else\addcytoks{#1}\fi}
\newcommand\apply[1]{\tcpopliteral\V\addcytoks{\ensuremath}%
  \groupedcytoks{\addcytoks{#1}\addcytoks[1]{\V}}}
\begin{document}
\tokencyclexpress \vec a,b,\vec{cx},d\endtokencyclexpress

\detokenize\expandafter{\the\cytoks}
\end{document}