# Parsing a mix of plain text and commands

I am trying to implement a command that parses some expression that can be a mix of plain text, commands with their (any number of) arguments, or even -- if I understood that correctly -- unexpandable parts.

To the end of parsing, I wish to scan the input 'item' by 'item', where an item is either a single char, maybe a {} pair, any command along with its arguments, or a single unexpandable stuff.

My first attempt was to recursively scan the input with :

\NewDocumentCommand\scan{m u{\relax}}{%
#1%
\ifblank{#2}
{}{,\scan#2\relax}%
}


that renders a,b,c,d for \scan abcd\relax. However, commands must be wrapped for them to work, as in : \scan a{\vec x}bcd\relax, which correctly renders a,x,b,c,d (where x has a vector arrow). Depending on how they are defined, even nullary commands need wrapping. This requirement is very annoying to me so I wish to get rid of it. Furthermore, a wrapped {⋅⋅⋅} would eventually need recursive parsing so protecting commands with it is ultimately not acceptable since it wouldn't fix the issue anymore.

My second attempt was to use xstring on the input, unfortunately it only works with plain text.

My third attempt was to detokenize/retokenize. But I can't find how to make \tokenize work, and didn't find documentation on it : \tokenize{\detokenize{⋅⋅⋅}} for instance yields unrelated errors. Also I'm not convinced this would lead to a practical solution since that would leave me with the problem of grabbing commands with all their arguments, not mentionning the potential trouble of spaces removal.

Finally, I went back to the first attempt but with \expandafter in the hope of transforming a command and its arguments into a unique 'solid' thing, as in :

\NewDocumentCommand\scan{m u{\relax}}{%
#1%
\ifblank{#2}
{}{,\expandafter\scan#2\relax}%
}
\NewDocumentCommand\scanit{m}{%
\expandafter\scan#1\relax%
}


Which I can't get to compile when input contains commands, for instance I would like $$scanit {\vec abcd}$$ to render a,b,c,d (where a has a vector arrow), but I get : Extra }, or forgotten $. $$\scanit {\vec abcd}$$. In fact, there are cases where it doesn't work but it would without \expandafter -- e.g. with some nullaries. Obviously I'm not in control of everything i'm doing here, so any explanation is appreciated • Welcome to TeX.SE. – Mico Mar 24 '21 at 14:06 • You gave abcd and a\vec{b}cd as use cases, which should be rendered as a, b, c, d and a, \vec{b}, c, d. Are groups of single letters and macros the only possible use cases? If not, please provide some additional examples. – Mico Mar 24 '21 at 14:10 • @Mico thank you, I tried to clarify a bit. I think the answer to your question is yes : I want to split single letters or single anything-that-should-come-as-one-thing Mar 24 '21 at 14:53 • @Planted I tried you first code and your edit, I still get Use of \next doesn't match its definition. \scanit{a{$\vec b\$}cde}. Bytheway, my goal is to not have to wrap the vec b Mar 24 '21 at 14:55
• @kmft3kte whoops, I misread the question. I have no idea how to do it then. Mar 24 '21 at 15:00

It is not exactly clear what you are trying to achieve, but the tokcycle package is set up to go through each token of an input stream and process them according to user defined directives. Tokens and braced material are categorized as Characters, Groups, Macros, or Spaces (CGMS), each of which gets it own directive.

While it can be told to stream the output directly as the input is processed, this only works for simple character input streams. If macros are involved, you can't execute the macro until its arguments have been processed. Thus, the typical and recommended mode of operation is to process the input and collect the processed tokens in a provided token list called \cytoks. The calls to \addcytoks are calls to add processed output to the token list. Then, once the end of the environment is reached, the processed \cytoks token list is rendered.

The #1 in the directives refers to the "current token" being processed. Throughout the environment, all tokens of a given type C, G, M, or S will successively go through their same respective directive and be processed according to the user defined directions. The default directives merely echo the input unchanged to the \cytoks list.

See the documentation and examples document at https://ctan.org/pkg/tokcycle

EDITED to handle regular braced-argument syntax.

\documentclass{article}
\usepackage{tokcycle}
\newif\ifvecnext
\stripgroupingtrue
\Characterdirective{\tctestifcon\ifvecnext
\vecnextfalse
}
\Groupdirective{\tctestifcon\ifvecnext
{\groupedcytoks{\processtoks{#1}}}%
\vecnextfalse
}
\Macrodirective{\tctestifx{\vec#1}
{\vecnexttrue}%
}\begin{document}
\tokencyclexpress \vec a,b,\vec{cx},d\endtokencyclexpress
\end{document}


To create your own named tokcycle environment, there is this:

\documentclass{article}
\usepackage{tokcycle}
\newif\ifvecnext
\stripgroupingtrue
\tokcycleenvironment\myscan% NEW TOKCYCLE ENVIRONMENT
{\tctestifcon\ifvecnext
\vecnextfalse
}% ↑ CHARACTER DIRECTIVE
{\tctestifcon\ifvecnext
{\groupedcytoks{\processtoks{##1}}}%
\vecnextfalse
}% ↑ GROUP DIRECTIVE
{\tctestifx{\vec##1}
{\vecnexttrue}%
}% ↑ MACRO DIRECTIVE
\begin{document}
\myscan \vec a,b,\vec{cx},d\endmyscan
\end{document}


SUPPLEMENT

As described in the SUPPLEMENT to this answer, LaTeX3: Elegant way to forward-reference a counter with a "future" value, I am working on package improvements to allow one to have the tokcycle directives access the future input stream directly. This will allow one to avoid the process of setting a flag in the Macro directive that is monitored by the Character directive. (EDIT: tokcycle[2021-05-27] has been released, so \makeatletter code has been removed from this MWE)

I envision the ability to pop and push tokens from the input stream. You can see that I avoid the Character directive altogether. In the \apply macro, I make use of two interesting package macros, \groupedcytoks and \tcpopliteral. Read all about them in the package documentation at https://ctan.org/pkg/tokcycle. Here, \groupedcytoks is used to introduce an additional group level into the output stream, namely, the argument to \ensuremath. The macro \tcpopliteral is used to pop an argument from the input stream, while preserving any leading spaces and grouping of the popped tokens (unlike normal TeX absorption that strips leading space as well as cat-1,2 tokens from the absorption of a group).

\documentclass{article}
\usepackage{tokcycle}

• Thank you, your package looks impressive. I would need to dive in a bit more, but I don't see quite what I want in the examples. The code you posted here does not correspond to my question because you specifically expect the command \vec, where I wish to handle any command whatsoever. I also want to respect the commands' arguments, therefore not considering them for parsing : abc\vec{xy}d for instance, as input, would become a,b,c,xy,d (with an arrow above xy) Mar 25 '21 at 9:46
• that's even trickier : What about commands that takes special arguments like e{^_}, or "wierd" arguments... Ideally, it would recursively process the next wathever into an unexpandable bit, so then I can take it Mar 25 '21 at 10:30
• @kmft3kte That is what the Macro directive is intended for. You have to therein specify all the conceivable cases you wish to cover. You can use the \ifx\macro-name#1 approach to shunt handling off to sub-directives that you write on how to handle processing of that \macro-name. You can include multiple \ifx handlers in the macro directive. In the example I provided, \apply handler might be able to handle any number of macros, such as \vec, \dot, etc. But you would have to direct them there accordingly. Mar 25 '21 at 11:07
• @kmft3kte To expand further, I have set the default behavior in the Macro directive to not alter the syntax, and have the user enumerate the cases needing special attention. An alternative would be to send, by default, all macros to \apply, but then you could have to enumerate all cases where require alternate treatment. There is no easy fix here...you are asking to change the TeX syntax for argument absorption. Your fix will have to be exhaustive. BTW, the approach in my supplement will honor the \vec{ax} syntax appropriately. Mar 25 '21 at 12:53