3

I am interesting in parsing a string of input character by character, and doing something to each character. In this MWE, I merely apply a \textbf to each successive character as an example, to verify that the parser is working.

The problem arises if the argument contains embedded macros. What I would like to do, if I come a across a macro in the argument, is to halt the parsing and let the macro execute, eating up as many arguments as it would desire from the original argument stream, and then resume parsing the remainder of the argument.

While \charparse is the parsing macro, I am trying to design a helper macro \execmacro to do what needs to occur when a macro is detected in the argument stream. In the MWE below, I can accomplish that, but only if I presuppose the nature of the embedded macros. In particular, I show 3 versions of \execmacro, depending on whether I presuppose the embedded macros to require 0, 1, or 2 arguments, respectively.

What I would like instead is to have a version of \execmacro that will work regardless of how many arguments the embedded macros demand.

\documentclass{article}
\newcommand\charparse[1]{\charparsehelp#1\relax\relax\relax}
\def\charparsehelp#1#2\relax{%
  \ifcat\noexpand\relax\noexpand#1%
%   MACRO DETECTED IN INPUT STREAM
    \execmacro#1#2\relax% EXECUTE THE MACRO THEN RETURN TO PARSING 
  \else\textbf{#1}% BOLDING IS JUST AN EXAMPLE TO SHOW THAT PARSING MACRO IS WORKING
  \ifx\relax#2\else
    \charparsehelp#2\relax
  \fi\fi
}
\begin{document}
% THIS ONLY WORKS IF THE EMBEDDED MACRO TAKES EXACTLY ZERO ARGUMENTS
\def\execmacro#1#2\relax{#1\ifx\relax#2\else\charparsehelp#2\relax\fi}Case 1\par
\charparse{0123\itshape456\upshape789}\par
I would like the output from the above character-parsing macro to be\par
\charparse{0123}\itshape\charparse{456}\upshape\charparse{789}\medskip

% THIS ONLY WORKS IF THE EMBEDDED MACRO TAKES EXACTLY ONE ARGUMENT
\def\execmacro#1#2#3\relax{#1{#2}\ifx\relax#3\else\charparsehelp#3\relax\fi}Case 2\par
\charparse{0123\textit{456}789}\par
I would like the output from the above character-parsing macro to be\par
\charparse{0123}\textit{456}\charparse{789}\medskip

% THIS ONLY WORKS IF THE EMBEDDED MACRO TAKES EXACTLY TWO ARGUMENTS
\def\execmacro#1#2#3#4\relax{#1{#2}{#3}\ifx\relax#4\else\charparsehelp#4\relax\fi}Case 3\par
\charparse{0123\rule{3ex}{1ex}456789}\par
I would like the output from the above character-parsing macro to be\par
\charparse{0123}\rule{3ex}{1ex}\charparse{456789}
\end{document}

enter image description here

Improve this question
9
  • You surely know that a macro such as \makebox has no argument: you'd need to build a TeX parser.
    – egreg
    Oct 7, 2015 at 14:12
  • @egreg I do not understand what your comment is trying to tell me.
    – Steven B. Segletes
    Oct 7, 2015 at 14:20
  • You can't decide the action of a macro just based on the number of its arguments.
    – egreg
    Oct 7, 2015 at 14:22
  • @egreg So that I understand you, you mean that a macros actions can chew up more arguments than are nominally called for.
    – Steven B. Segletes
    Oct 7, 2015 at 14:24
  • @egreg I understand that, in advance, one in general does not know how much of the argument stream will be eaten by the macro. What I was hoping is that a way exists to temporarily redefine the macro to reinvoke parsing as its final dying act.
    – Steven B. Segletes
    Oct 7, 2015 at 14:29

2 Answers 2

Reset to default
2

Well, since no one wants to touch this one, and all the gurus advise strongly against it, I should probably let it lie. But I won't. Just to show what might be done, I followed the hint mentioned by egreg, to the effect of "registering" allowable macros that can be processed.

At this time, the presence of optional arguments are not allowed, which is something of a drawback. And for demonstration purposes, I have set it up to handle macros with up to 2 arguments only.

So here is how it works. I wish to develop \charparse{} that parses the argument 1 character at a time. But the key is to be able to execute macros in the argument and then resume the character by character parsing following their execution.

If a macro has no arguments, it should not be registered. Otherwise, the macros have to be registered with \registerparsemacro}{\<macroname>}{<arguments>}. To repeat, optional arguments are not allowed. So, for instance, I invoke

\registerparsemacro{\textit}{1}
\registerparsemacro{\rule}{2}

At that point I can include invocations of \textit and \rule (without optional argument) in the argument to \charparse. The macros \itshape and \upshape should not be registered, since they take no arguments.

The following MWE creates the same output as the question shows.

\documentclass{article}
\usepackage{ifthen}
\newcommand\charparse[1]{\charparsehelp#1\relax\relax\relax}
\def\charparsehelp#1#2\relax{%
  \ifcat\noexpand\relax\noexpand#1%
%   MACRO DETECTED IN INPUT STREAM
    \execmacro#1#2\relax% EXECUTE THE MACRO THEN RETURN TO PARSING 
  \else\textbf{#1}% BOLDING IS JUST AN EXAMPLE TO SHOW THAT PARSING MACRO IS WORKING
  \ifx\relax#2\else
    \charparsehelp#2\relax
  \fi\fi
}
\newcounter{parsemacro}
\def\execmacro#1#2#3#4\relax{%
  \setcounter{parsemacro}{0}%
  \whiledo{\value{parsemacro} < \value{parsemacrocount}}{%
  \stepcounter{parsemacro}%
  \expandafter\expandafter\expandafter%
    \ifx\csname parsemacro\romannumeral\value{parsemacro}\endcsname#1%
      \if1\csname parsemacroarguments\romannumeral\value{parsemacro}\endcsname
        \execmacroONE#1{#2}#3#4\relax\else
      \if2\csname parsemacroarguments\romannumeral\value{parsemacro}\endcsname
        \execmacroTWO#1{#2}{#3}#4\relax\else
      \fi\fi
      \setcounter{parsemacro}{\numexpr\value{parsemacrocount}+1}%
    \fi
  }%
  \ifnum\value{parsemacro}=\value{parsemacrocount}\execmacroZERO#1#2#3#4\relax\fi
}
\def\execmacroZERO#1#2\relax{#1\ifx\relax#2\else
  \charparsehelp#2\relax\fi}
\def\execmacroONE#1#2#3\relax{#1{#2}\ifx\relax#3\else
  \charparsehelp#3\relax\fi}
\def\execmacroTWO#1#2#3#4\relax{#1{#2}{#3}\ifx\relax#4\else
  \charparsehelp#4\relax\fi}
\newcounter{parsemacrocount}
\setcounter{parsemacrocount}{0}
\def\registerparsemacro#1#2{%
  \stepcounter{parsemacrocount}%
  \expandafter\def\csname parsemacro\romannumeral\value{parsemacrocount}\endcsname{#1}%
  \expandafter\def\csname parsemacroarguments\romannumeral\value{parsemacrocount}%
    \endcsname{#2}%
}
\begin{document}
\registerparsemacro{\textit}{1}
\registerparsemacro{\rule}{2}
Case 1\par\charparse{0123\itshape456\upshape789}\par
I would like the output from the above character-parsing macro to be\par
\charparse{0123}\itshape\charparse{456}\upshape\charparse{789}\medskip

Case 2\par\charparse{0123\textit{456}789}\par
I would like the output from the above character-parsing macro to be\par
\charparse{0123}\textit{456}\charparse{789}\medskip

Case 3\par\charparse{0123\rule{3ex}{1ex}456789}\par
I would like the output from the above character-parsing macro to be\par
\charparse{0123}\rule{3ex}{1ex}\charparse{456789}
\end{document}
Improve this answer
3
  • I would just step over all commands and brace groups then you don't need to list most commands \foo{zz} could be skipped (if you don't mind skipping {abc} but * forms and optional arguments are pretty hard to deal with something like \section*[abc]{yyz}` the *[abc] is just character data unless you know the operational behaviour of \section
    – David Carlisle
    Oct 7, 2015 at 20:06
  • @DavidCarlisle Thanks for the idea of skipping over brace groups. To go that route would allow the avoidance of registering each macro anticipated. But I would still need to execute them as I am skipping over them.
    – Steven B. Segletes
    Oct 7, 2015 at 20:10
  • No I'd just gather up all of the original argument into a token register, characters you split up and add to the token register as \foo{x} csnames and brace groups just add to the register as-is, then at the end do \the\toks@ to play it all through,
    – David Carlisle
    Oct 7, 2015 at 20:14
1

Nearly 5 years after asking the question, I realize my recent tokcycle package can answer this question, directly. I have even made it to handle optional arguments, which it assumes to occur when a [ follows a control sequence. (Thus, to place a normal [ following a control sequence, use \macro{}[ in the input stream, placing empty braces following the control sequence.)

Here it does it by creating the \Charparse macro that leaves control sequences, spaces, and (importantly) group content untouched, thus operating only upon top-level character tokens. Additional screening could be added to restrict operations to particular catcodes, or to exclude particular character tokens.

Thus, as long as arguments to a macro are embraced {}, they will be passed through unmodified.

In the MWE below, the first 3 examples are from the OP's question. The 4th example shows an optional argument being handled properly. The last example shows how to overcome a normal [ accidentally being interpreted as an optional argument.

\documentclass{article}
\usepackage{tokcycle}
\newcommand\Charparse[1]{%
  \def\macON{F}%
  \def\optON{F}%
  \tokcycle
    {\if T\macON\ifx[##1\def\optON{T}\fi\fi
     \if T\optON\addcytoks{##1}\else\addcytoks{\textbf{##1}}\fi
     \ifx]##1\def\optON{F}\fi
     \def\macON{F}%
    }% CHARACTER DIRECTIVE
    {\addcytoks{##1}\gdef\macON{F}}% GROUP DIRECTIVE
    {\addcytoks{##1}\if F\optON\def\macON{T}\fi}% MACRO (CS) DIRECTIVE
    {\addcytoks{##1}\def\macON{F}}% SPACE DIRECTIVE
  {#1}\the\cytoks
}
\begin{document}
Tokcycle can define an environment that leaves\\
 macros, spaces, and group content untouched.\par
\Charparse{
0123\itshape456\upshape789

0123\textit{456}789

0123\rule{3ex}{1ex}456789

0123\rule[-3pt]{3ex}{1ex}456789

\S[x] \S{}[x]}
\end{document}

enter image description here

For those wondering, tokcycle can be directed to process tokens inside of groups (using \processtoks{##1} instead of \addcytoks{##1} for the Group Directive). However, in this application, it would defeat the purpose

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.