What is the easiest way to modify catcodes in a string?

Question

Assume that a token register contains a sequence of letter tokens, each having a catcode of 12. I want to store in a second token register the same sequence of letter token, altering some catcodes on the way (like changing all # so that they have catcode 6).

I see three ways to do this:

1. Iterating over the sequence of tokens with futurelet and do the replacement. This method has the inconvenient that adding tokens at the end of a token register involves a lot of expansions and parsing but the advantage that it can be generalised.

2. Using a macro with an argument template (like what we do to see if a token is in a sequence) to quickly parse the token register. It has the drawback that it is maybe not easy to generalise.

3. Write the tokens in some auxiliary file, reconfigure the lexer and reread these tokens. It is ugly.

Is there an easier way to go? (In plain TeX, no Lua, no Perl, no OCaml…)

Answer 1

I took this question as a learning opportunity for me, more than an actual attempt to satisfy a question for you, and so my answer is not yet a generalized approach (though I think it could get there with a little work). I would say this method falls in your category #2, "Using a macro with an argument template (like what we do to see if a token is in a sequence) to quickly parse the token register."

I didn't know how to test whether I was successful with manipulating a # symbol's catcode, so I took a less challenging problem, the underscore _, knowing that I could throw it between some dollar signs and immediately know whether I was successful in changing the catcode.

The solution was EDITED so that the \iterationengine is now recursive, and will continue until all catcode12 underscores have been replaced with catcode8 underscores. Originally, I had to invoke the iteration engine successively, by hand.

In the MWE below, after setting up my macro, I test it on four separate strings, containing 0, 1, 2, and 3 underscores.

I haven't been able to confirm it, but the detector routine (\detectus) may fail if it encounters a catcode12 underscore followed by a normal catcoded \testchar (in this case, \relax).

\documentclass{article}
\usepackage[T1]{fontenc}
\def\testchar{\relax}
\def\usEight{_}
\catcode`_=12
\def\detectus#1_#2\relax{%
  \def\Detected{#2\testchar}\if\testchar\Detected\else\def\Detected{T}\fi}
\def\changeus#1_#2\relax{#1\usEight#2}
\makeatletter\def\iterationengine{%
  \if T\Detected%
    \expandafter\detectus\catEightTok_\relax%
    \if T\Detected\protected@edef\catEightTok{\expandafter\changeus\catEightTok\relax}\fi%
    \iterationengine%
  \fi%
}\makeatother
\catcode`_=8
\def\iteration{\let\catEightTok\catTwelveTok\def\Detected{T}\iterationengine}
\long\def\runtest{%
  \makebox[3cm][l]{$\rawtok$} raw token in math\par
  \edef\catTwelveTok{\detokenize\expandafter{\rawtok}}
  \makebox[3cm][l]{\catTwelveTok} detokenized raw token (cat12)\par
  \makebox[3cm][l]{$\catTwelveTok$} cat12 token in math\par
  \iteration
  \makebox[3cm][l]{\detokenize\expandafter{\catEightTok}} 
    detokenized converted token (cat 8)\par
  \makebox[3cm][l]{$\catEightTok$} converted token, in math
}
\begin{document}
\vspace{1ex}\def\rawtok{ab}\runtest\par
\vspace{1ex}\def\rawtok{a_b}\runtest\par
\vspace{1ex}\def\rawtok{a_b c_d}\runtest\par
\vspace{1ex}\def\rawtok{a_b c_d e_f}\runtest\par
\end{document}

Answer 2

This is a Plain TeX (or any other format) general purpose macro \scanthechars which accepts on input a string of character tokens of catcode 12 (1) and puts them in the same order but with transformed catcodes in a token list register called \replacetoks.

(1) this update adds two \string's to the \@@scanthechars macro, and as a result the input is not anymore restricted to be with catcode 12 tokens only. It may even contain control sequences (but not braced material) which will go through unarmed, and one may convert back and forth between various catcode regimes -- see updated image.

The interface to specify the catcodes is via a macro \replacesetup which is fetched with a comma separated list of things like \!{3} which means to transform the character token ! into a catcode 3 character token !.

The illustrative tests use \detokenize for simplicity sake, to produce easily catcode 12 tokens, or verbatim output, hence the code needs etex or pdftex for compilation. But the macros themselves are strictly Knuthian.

Only catcodes of 3, 4, 6, 7, 8, 11, 12, 13 are dealt with.

Each use of \scanthechars resets \replacesetup: next call to \scanthechars must follow a renewed \replacesetup. Since now the input is not restricted to be only with catcode 12 tokens, one may use \scanthechars many times on the same string. See the code and image for how this is done.

---

The code:

\catcode`@ 11
\long\def\@gobble      #1{}
\long\def\@firstoftwo  #1#2{#1}
\long\def\@secondoftwo #1#2{#2}

\newtoks\replacetoks

\def\replacesetup #1{%
% this assumes non nil escapechar
    \def\replace@list {}%
    \def\replace@do ##1##2%
       {\expandafter\def\csname replace@setup@\expandafter
                                      \@gobble\string##1\endcsname {##2}}%
    \replace@setup #1,\relax\relax,%
}

\def\replace@setup #1#2,{%
    \ifx#1\relax\replace@list
    \else
        \expandafter\def\expandafter\replace@list\expandafter
          {\replace@list \replace@do #1{#2}}%
        \expandafter\replace@setup
    \fi
} 

\def\scanthechars #1{\replacetoks{}%
           \edef\replace@restore{\lccode`$=\the\lccode`$
                                 \lccode`^=\the\lccode`^
                                 \lccode`_=\the\lccode`_
                                 \lccode`&=\the\lccode`&
                                 \lccode`\noexpand\#=\the\lccode`\#
                                 \lccode`a=`a \lccode`(=\the\lccode`(
                                 \lccode`\noexpand~=\the\lccode`~\relax
                                 }%
           \@scanthechars #1\relax}


\def\@scanthechars #1{\ifx #1\relax\expandafter\@firstoftwo
                              \else\expandafter\@secondoftwo
                      \fi
                      {\def\replace@do ##1##2{\expandafter
                         \let\csname replace@setup@\expandafter
                                     \@gobble\string##1\endcsname\relax}%
                       \replace@list % resets to relax everything
                       \replace@restore}%
                      {\@@scanthechars #1}%
}

\def\@@scanthechars #1{\expandafter
                 \ifx\csname replace@setup@\string#1\endcsname\relax
                    \replacetoks\expandafter{\the\replacetoks #1}%
                 \else
                    \ifcase\csname replace@setup@\string#1\endcsname\relax
                    \or\or\or % catcode=3
                      \lccode`$=`#1 %$
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks $}}%$
                    \or % catcode=4
                      \lccode`&=`#1 
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks &}}%
                    \or\or % 6
                      \lccode`\#=`#1 
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks ##}}%
                    \or % 7
                      \lccode`^=`#1 
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks ^}}%
                    \or % 8
                      \lccode`_=`#1 
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks _}}%
                    \or\or\or % 11
                       \lccode`a=`#1 
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks a}}%
                    \or % 12
                       \lccode`(=`#1 
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks (}}%
                    \or % 13
                       \lccode`~=`#1
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks ~}}%
                   \fi
                \fi
    \@scanthechars
}

\catcode`@ 12
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% utility macro to get the catcode of a character token
% manages only catcodes 3,4,6,7,8,11,12,13
\def\TheCatcode #1{\ifcat\noexpand#1$3\else% $
                    \ifcat\noexpand#1&4\else
                     \ifcat\noexpand#1##6\else
                      \ifcat\noexpand#1^7\else
                       \ifcat\noexpand#1_8\else
                        \ifcat\noexpand#1a11\else
                         \ifcat\noexpand#1(12\else
                          \ifcat\noexpand#1\noexpand~13\else not handled
                   \fi\fi\fi\fi\fi\fi\fi\fi }

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\tt

Test I. 

\replacesetup{\#{3},\!{7},\?{8}}

\detokenize{\replacesetup{\#{3},\!{7},\?{8}}}

\edef\x{\string#a!b?c\string#}  

\x

\expandafter\scanthechars\expandafter{\x}

\the\replacetoks

\begingroup
  \catcode`#=3
  \catcode`!=7
  \catcode`?=8
  \gdef\z{#a!b?c#}
\endgroup

\the\replacetoks
=\z

\edef\y{\the\replacetoks}

\ifx\z\y OK\else WRONG\fi

\bigskip

Test II.

\replacesetup{\a{11},\b{11},\c{11},\\{3}}

\detokenize{\replacesetup{\a{11},\b{11},\c{11},\\{3}}}

\catcode`@ 11
\edef\backslashchar{\expandafter\@gobble\string\\}
\catcode`@ 12

% creates an x with catcode 12 a, b, c
\edef\x{\string\abc\backslashchar}

\x

\expandafter\scanthechars\expandafter{\x}

\edef\y{\the\replacetoks}

\begingroup
\catcode`|=0
\catcode`\\=3
|gdef|z{\abc\}
|endgroup

\y=\z

\ifx\z\y OK\else WRONG\fi


\bigskip

Test III.

\replacesetup{\a{3},\b{7},\c{8},\d{12}}

\detokenize{\replacesetup{\a{3},\b{7},\c{8},\d{12}}}

\expandafter\scanthechars\expandafter {\detokenize{abcd}}% convert to catcode 12

\edef\y {\the\replacetoks}

\def\PrintTheCatcode #1{\string#1 with catcode \TheCatcode #1\par }
% previously, we used xinttools.sty:
% \xintApplyInline{\PrintTheCatcode}{\y}

\def\PrintTheCatcodes #1{\ifx#1\relax\else\PrintTheCatcode#1\expandafter
                                          \PrintTheCatcodes\fi }
\expandafter\PrintTheCatcodes\y\relax

% \replacesetup must be redone each time
\replacesetup{\a{3},\b{7},\c{8},\d{12}}

\expandafter\scanthechars\expandafter {\detokenize{adbdcda}}

\detokenize{adbdcda}->\the\replacetoks

\bigskip

Test IV.

\overfullrule 0pt

\replacesetup {\!{6}}

\scanthechars {!}

\expandafter\meaning\the\replacetoks

\edef\y{\the\replacetoks}

\begingroup
\catcode`!6
\gdef\z{!!}
\endgroup

\ifx\z\y OK\else WRONG\fi

\bigskip
TEST V.

From now on we don't require the input to be with catcode 12 tokens.

\replacesetup{\#{3}, \_{11}, \^{11}, \!{4}, \?{7}, \&{8}, \@{6}}

\detokenize{\replacesetup{\#{3}, \_{11}, \^{11}, \!{4}, \?{7}, \&{8}, \@{6}}}

\scanthechars {\tabskip1ex #\mathstrut @#\hfil!\hfil #@#\hfil!\hfil#@#\cr
               \noalign\bgroup\hrule\egroup
                        1!23!456\cr 7890!a!bc\cr
                \noalign\bgroup\hrule\egroup }

\detokenize{\scanthechars {\tabskip1ex #\mathstrut @#\hfil!\hfil #@#\hfil!\hfil#@#\cr
               \noalign\bgroup\hrule\egroup
                        1!23!456\cr 7890!a!bc\cr
                \noalign\bgroup\hrule\egroup }

}


\detokenize{\halign{\span\the\replacetoks}}

output:
\medskip
\halign{\span\the\replacetoks}
% \expandafter\PrintTheCatcodes\the\replacetoks\relax

% attention _ is mathematically active!
% in plain.tex \mathcode`\_="8000 % \_

\bigskip


\replacesetup{\#{3}, \_{11}, \^{11}, \!{4}, \?{7}, \&{8}, \@{6}}
\detokenize {\replacesetup{\#{3}, \_{11}, \^{11}, \!{4}, \?{7}, \&{8}, \@{6}}}
\scanthechars {##___^?\bgroup abc\egroup &\bgroup\hbox\bgroup___\egroup\egroup##}

\detokenize {\scanthechars {##___^?\bgroup abc\egroup &\bgroup\hbox\bgroup___\egroup\egroup##}}

\detokenize{\hrule\the\replacetoks\hrule}

output:

\hrule\the\replacetoks\hrule

\medskip
\string_ is mathematically active, the subscript is in an \string\hbox\space
hence the catcode 11 version is used, but the first three use \string\_ as
defined in plain.tex for the math active \string_, which now has catcode letter
\bigskip

Let's now go back to catcode 12:

\edef\y{\the\replacetoks}

\replacesetup {\#{12}, \_{12}, \^{12}, \!{12}, \?{12}, \&{12}, \@{12}}

\expandafter\scanthechars\expandafter{\y}

\detokenize{\edef\y{\the\replacetoks}}

\detokenize{\replacesetup {\#{12}, \_{12}, \^{12}, \!{12}, \?{12}, \&{12}, \@{12}}}

\detokenize{\expandafter\scanthechars\expandafter{\y}}

\detokenize{\the\replacetoks}


\the\replacetoks

\edef\z{\the\replacetoks}

\detokenize{\edef\z{\the\replacetoks}\meaning\z}

\meaning\z
\nopagenumbers

\bye 

Answer 3

In the intervening years since my other answer, I have written the tokcycle package for taking an argument or input stream one token at a time, and "doing something with it". It can easily be set up to do something with cat-12 # tokens and reconvert them to cat-6, since this seems to be what the OP desires.

The MWE works through a progression:

1. the standard cat-6 approach to \def

2. What happens if the argument of the \def contains cat-12 #.

3. What happens if both the argument and the \def parameter itself are cat-12 #.

4. Now I set up tokcycle to convert cat-12 # tokens back to cat-6. In this step, I first pass only cat-6 # tokens to the cycle, to show that it can process them without issue (but no conversion is needed).

5. Here, I convert a cat-12 # in the argument to a \def back to cat-6, with the expected result.

6. Here, I convert the cat-12 # tokens in both the argument and in the \def parameter specification back to cat-6, with the expected result.

7. Here, I operate in an underlying cat-12 # environment, and still successfully convert occurrences of # back to cat-6, with the expected result.

The MWE:

\documentclass {article}
\usepackage[T1]{fontenc}
\usepackage{tokcycle}
\let\SIXhash#
\catcode`#=12
\let\TWELVEhash#
\catcode`#=6
\begin{document}
1.\def\Q#1{My argument is [#1]}
\Q{X}

2.\def\QQ#1{My argument is [\TWELVEhash1]}
\QQ{X}

3.\def\QQQ\TWELVEhash1{My argument is [\TWELVEhash1]}
\QQQ\TWELVEhash1

\Characterdirective{%
  \whennotprocessingparameter#1{\tctestifx{\TWELVEhash#1}%
    {\addcytoks{##}}{\addcytoks{#1}}}}

4.\tokencyclexpress
\gdef\W#1{My argument is [#1]}
\endtokencyclexpress
\W{X}

5.\tokencyclexpress
\gdef\WW#1{My argument is [\TWELVEhash1]}
\endtokencyclexpress
\WW{X}

6.\tokencyclexpress
\gdef\WWW\TWELVEhash1{My argument is [\TWELVEhash1]}
\endtokencyclexpress
\WWW{X}

7.\catcode`#=12
\tokencyclexpress
\gdef\WWWW#1{My argument is [#1]}
\endtokencyclexpress
\WWWW{X} #####
\end{document}

---

SUPPLEMENT

The case, above, of cat-6 changes is more challenging because of the way cat-6 tokens are processed (the need for ##, ####, etc). For other catcode conversions, the tokcycle process is easier, since I don't have to ascertain whether the token is part of a parameter. Here, I do something analogous with the underscore _.

\documentclass {article}
\usepackage[T1]{fontenc}
\usepackage{tokcycle}
\catcode`_=12
\let\TWELVEus_
\catcode`_=8
\begin{document}
1.\def\Q{$a\TWELVEus1$}
\Q

\Characterdirective{%
  \tctestifx{\TWELVEus#1}{\addcytoks{_}}{\addcytoks{#1}}}

2. \expandafter\tokencyclexpress\Q\endtokencyclexpress

3.\catcode`_=12
\def\Q{$a_1$}
\expandafter\tokencyclexpress\Q\endtokencyclexpress

4. $\tokencyclexpress a_1\endtokencyclexpress$
____
\end{document}