15

Assume that a token register contains a sequence of letter tokens, each having a catcode of 12. I want to store in a second token register the same sequence of letter token, altering some catcodes on the way (like changing all # so that they have catcode 6).

I see three ways to do this:

  1. Iterating over the sequence of tokens with futurelet and do the replacement. This method has the inconvenient that adding tokens at the end of a token register involves a lot of expansions and parsing but the advantage that it can be generalised.

  2. Using a macro with an argument template (like what we do to see if a token is in a sequence) to quickly parse the token register. It has the drawback that it is maybe not easy to generalise.

  3. Write the tokens in some auxiliary file, reconfigure the lexer and reread these tokens. It is ugly.

Is there an easier way to go? (In plain TeX, no Lua, no Perl, no OCaml…)

  • 3
    Is e-TeX's \scantokens allowed? – egreg Dec 5 '13 at 18:22
  • @egreg No it is not. :) I am looking for a TeXbook answer. – Michael Le Barbier Grünewald Dec 5 '13 at 18:23
  • 4
    So the only way is a slow parsing character by character. – egreg Dec 5 '13 at 18:48
  • 3
    Refusing e-TeX is really making it deliberately hard, in contradiction to the question title. Is this for a paper on the history of TeX? The TeXbook answer is really "Don't do this kind of thing. TeX was not made for it." – Stephan Lehmke Dec 8 '13 at 6:00
  • 1
    are there space tokens possibly involved or exclusively catcode 12 (perhaps 11?) character tokens? (I am asking this because \string or e-TeX \detokenize may produce catcode 10 spaces, and perhaps this is what you have as input to your catcode changing task). – user4686 Mar 13 '14 at 18:51
7

I took this question as a learning opportunity for me, more than an actual attempt to satisfy a question for you, and so my answer is not yet a generalized approach (though I think it could get there with a little work). I would say this method falls in your category #2, "Using a macro with an argument template (like what we do to see if a token is in a sequence) to quickly parse the token register."

I didn't know how to test whether I was successful with manipulating a # symbol's catcode, so I took a less challenging problem, the underscore _, knowing that I could throw it between some dollar signs and immediately know whether I was successful in changing the catcode.

The solution was EDITED so that the \iterationengine is now recursive, and will continue until all catcode12 underscores have been replaced with catcode8 underscores. Originally, I had to invoke the iteration engine successively, by hand.

In the MWE below, after setting up my macro, I test it on four separate strings, containing 0, 1, 2, and 3 underscores.

I haven't been able to confirm it, but the detector routine (\detectus) may fail if it encounters a catcode12 underscore followed by a normal catcoded \testchar (in this case, \relax).

\documentclass{article}
\usepackage[T1]{fontenc}
\def\testchar{\relax}
\def\usEight{_}
\catcode`_=12
\def\detectus#1_#2\relax{%
  \def\Detected{#2\testchar}\if\testchar\Detected\else\def\Detected{T}\fi}
\def\changeus#1_#2\relax{#1\usEight#2}
\makeatletter\def\iterationengine{%
  \if T\Detected%
    \expandafter\detectus\catEightTok_\relax%
    \if T\Detected\protected@edef\catEightTok{\expandafter\changeus\catEightTok\relax}\fi%
    \iterationengine%
  \fi%
}\makeatother
\catcode`_=8
\def\iteration{\let\catEightTok\catTwelveTok\def\Detected{T}\iterationengine}
\long\def\runtest{%
  \makebox[3cm][l]{$\rawtok$} raw token in math\par
  \edef\catTwelveTok{\detokenize\expandafter{\rawtok}}
  \makebox[3cm][l]{\catTwelveTok} detokenized raw token (cat12)\par
  \makebox[3cm][l]{$\catTwelveTok$} cat12 token in math\par
  \iteration
  \makebox[3cm][l]{\detokenize\expandafter{\catEightTok}} 
    detokenized converted token (cat 8)\par
  \makebox[3cm][l]{$\catEightTok$} converted token, in math
}
\begin{document}
\vspace{1ex}\def\rawtok{ab}\runtest\par
\vspace{1ex}\def\rawtok{a_b}\runtest\par
\vspace{1ex}\def\rawtok{a_b c_d}\runtest\par
\vspace{1ex}\def\rawtok{a_b c_d e_f}\runtest\par
\end{document}

enter image description here

3

This is a Plain TeX (or any other format) general purpose macro \scanthechars which accepts on input a string of character tokens of catcode 12 (1) and puts them in the same order but with transformed catcodes in a token list register called \replacetoks.

(1) this update adds two \string's to the \@@scanthechars macro, and as a result the input is not anymore restricted to be with catcode 12 tokens only. It may even contain control sequences (but not braced material) which will go through unarmed, and one may convert back and forth between various catcode regimes -- see updated image.

The interface to specify the catcodes is via a macro \replacesetup which is fetched with a comma separated list of things like \!{3} which means to transform the character token ! into a catcode 3 character token !.

The illustrative tests use \detokenize for simplicity sake, to produce easily catcode 12 tokens, or verbatim output, hence the code needs etex or pdftex for compilation. But the macros themselves are strictly Knuthian.

Only catcodes of 3, 4, 6, 7, 8, 11, 12, 13 are dealt with.

Each use of \scanthechars resets \replacesetup: next call to \scanthechars must follow a renewed \replacesetup. Since now the input is not restricted to be only with catcode 12 tokens, one may use \scanthechars many times on the same string. See the code and image for how this is done.

catcode replace


catcode replace (seq.)

The code:

\catcode`@ 11
\long\def\@gobble      #1{}
\long\def\@firstoftwo  #1#2{#1}
\long\def\@secondoftwo #1#2{#2}

\newtoks\replacetoks

\def\replacesetup #1{%
% this assumes non nil escapechar
    \def\replace@list {}%
    \def\replace@do ##1##2%
       {\expandafter\def\csname replace@setup@\expandafter
                                      \@gobble\string##1\endcsname {##2}}%
    \replace@setup #1,\relax\relax,%
}

\def\replace@setup #1#2,{%
    \ifx#1\relax\replace@list
    \else
        \expandafter\def\expandafter\replace@list\expandafter
          {\replace@list \replace@do #1{#2}}%
        \expandafter\replace@setup
    \fi
} 

\def\scanthechars #1{\replacetoks{}%
           \edef\replace@restore{\lccode`$=\the\lccode`$
                                 \lccode`^=\the\lccode`^
                                 \lccode`_=\the\lccode`_
                                 \lccode`&=\the\lccode`&
                                 \lccode`\noexpand\#=\the\lccode`\#
                                 \lccode`a=`a \lccode`(=\the\lccode`(
                                 \lccode`\noexpand~=\the\lccode`~\relax
                                 }%
           \@scanthechars #1\relax}


\def\@scanthechars #1{\ifx #1\relax\expandafter\@firstoftwo
                              \else\expandafter\@secondoftwo
                      \fi
                      {\def\replace@do ##1##2{\expandafter
                         \let\csname replace@setup@\expandafter
                                     \@gobble\string##1\endcsname\relax}%
                       \replace@list % resets to relax everything
                       \replace@restore}%
                      {\@@scanthechars #1}%
}

\def\@@scanthechars #1{\expandafter
                 \ifx\csname replace@setup@\string#1\endcsname\relax
                    \replacetoks\expandafter{\the\replacetoks #1}%
                 \else
                    \ifcase\csname replace@setup@\string#1\endcsname\relax
                    \or\or\or % catcode=3
                      \lccode`$=`#1 %$
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks $}}%$
                    \or % catcode=4
                      \lccode`&=`#1 
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks &}}%
                    \or\or % 6
                      \lccode`\#=`#1 
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks ##}}%
                    \or % 7
                      \lccode`^=`#1 
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks ^}}%
                    \or % 8
                      \lccode`_=`#1 
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks _}}%
                    \or\or\or % 11
                       \lccode`a=`#1 
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks a}}%
                    \or % 12
                       \lccode`(=`#1 
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks (}}%
                    \or % 13
                       \lccode`~=`#1
                      \lowercase
                      {\replacetoks\expandafter{\the\replacetoks ~}}%
                   \fi
                \fi
    \@scanthechars
}

\catcode`@ 12
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% utility macro to get the catcode of a character token
% manages only catcodes 3,4,6,7,8,11,12,13
\def\TheCatcode #1{\ifcat\noexpand#1$3\else% $
                    \ifcat\noexpand#1&4\else
                     \ifcat\noexpand#1##6\else
                      \ifcat\noexpand#1^7\else
                       \ifcat\noexpand#1_8\else
                        \ifcat\noexpand#1a11\else
                         \ifcat\noexpand#1(12\else
                          \ifcat\noexpand#1\noexpand~13\else not handled
                   \fi\fi\fi\fi\fi\fi\fi\fi }

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\tt

Test I. 

\replacesetup{\#{3},\!{7},\?{8}}

\detokenize{\replacesetup{\#{3},\!{7},\?{8}}}

\edef\x{\string#a!b?c\string#}  

\x

\expandafter\scanthechars\expandafter{\x}

\the\replacetoks

\begingroup
  \catcode`#=3
  \catcode`!=7
  \catcode`?=8
  \gdef\z{#a!b?c#}
\endgroup

\the\replacetoks
=\z

\edef\y{\the\replacetoks}

\ifx\z\y OK\else WRONG\fi

\bigskip

Test II.

\replacesetup{\a{11},\b{11},\c{11},\\{3}}

\detokenize{\replacesetup{\a{11},\b{11},\c{11},\\{3}}}

\catcode`@ 11
\edef\backslashchar{\expandafter\@gobble\string\\}
\catcode`@ 12

% creates an x with catcode 12 a, b, c
\edef\x{\string\abc\backslashchar}

\x

\expandafter\scanthechars\expandafter{\x}

\edef\y{\the\replacetoks}

\begingroup
\catcode`|=0
\catcode`\\=3
|gdef|z{\abc\}
|endgroup

\y=\z

\ifx\z\y OK\else WRONG\fi


\bigskip

Test III.

\replacesetup{\a{3},\b{7},\c{8},\d{12}}

\detokenize{\replacesetup{\a{3},\b{7},\c{8},\d{12}}}

\expandafter\scanthechars\expandafter {\detokenize{abcd}}% convert to catcode 12

\edef\y {\the\replacetoks}

\def\PrintTheCatcode #1{\string#1 with catcode \TheCatcode #1\par }
% previously, we used xinttools.sty:
% \xintApplyInline{\PrintTheCatcode}{\y}

\def\PrintTheCatcodes #1{\ifx#1\relax\else\PrintTheCatcode#1\expandafter
                                          \PrintTheCatcodes\fi }
\expandafter\PrintTheCatcodes\y\relax

% \replacesetup must be redone each time
\replacesetup{\a{3},\b{7},\c{8},\d{12}}

\expandafter\scanthechars\expandafter {\detokenize{adbdcda}}

\detokenize{adbdcda}->\the\replacetoks

\bigskip

Test IV.

\overfullrule 0pt

\replacesetup {\!{6}}

\scanthechars {!}

\expandafter\meaning\the\replacetoks

\edef\y{\the\replacetoks}

\begingroup
\catcode`!6
\gdef\z{!!}
\endgroup

\ifx\z\y OK\else WRONG\fi

\bigskip
TEST V.

From now on we don't require the input to be with catcode 12 tokens.

\replacesetup{\#{3}, \_{11}, \^{11}, \!{4}, \?{7}, \&{8}, \@{6}}

\detokenize{\replacesetup{\#{3}, \_{11}, \^{11}, \!{4}, \?{7}, \&{8}, \@{6}}}

\scanthechars {\tabskip1ex #\mathstrut @#\hfil!\hfil #@#\hfil!\hfil#@#\cr
               \noalign\bgroup\hrule\egroup
                        1!23!456\cr 7890!a!bc\cr
                \noalign\bgroup\hrule\egroup }

\detokenize{\scanthechars {\tabskip1ex #\mathstrut @#\hfil!\hfil #@#\hfil!\hfil#@#\cr
               \noalign\bgroup\hrule\egroup
                        1!23!456\cr 7890!a!bc\cr
                \noalign\bgroup\hrule\egroup }

}


\detokenize{\halign{\span\the\replacetoks}}

output:
\medskip
\halign{\span\the\replacetoks}
% \expandafter\PrintTheCatcodes\the\replacetoks\relax

% attention _ is mathematically active!
% in plain.tex \mathcode`\_="8000 % \_

\bigskip


\replacesetup{\#{3}, \_{11}, \^{11}, \!{4}, \?{7}, \&{8}, \@{6}}
\detokenize {\replacesetup{\#{3}, \_{11}, \^{11}, \!{4}, \?{7}, \&{8}, \@{6}}}
\scanthechars {##___^?\bgroup abc\egroup &\bgroup\hbox\bgroup___\egroup\egroup##}

\detokenize {\scanthechars {##___^?\bgroup abc\egroup &\bgroup\hbox\bgroup___\egroup\egroup##}}

\detokenize{\hrule\the\replacetoks\hrule}

output:

\hrule\the\replacetoks\hrule

\medskip
\string_ is mathematically active, the subscript is in an \string\hbox\space
hence the catcode 11 version is used, but the first three use \string\_ as
defined in plain.tex for the math active \string_, which now has catcode letter
\bigskip

Let's now go back to catcode 12:

\edef\y{\the\replacetoks}

\replacesetup {\#{12}, \_{12}, \^{12}, \!{12}, \?{12}, \&{12}, \@{12}}

\expandafter\scanthechars\expandafter{\y}

\detokenize{\edef\y{\the\replacetoks}}

\detokenize{\replacesetup {\#{12}, \_{12}, \^{12}, \!{12}, \?{12}, \&{12}, \@{12}}}

\detokenize{\expandafter\scanthechars\expandafter{\y}}

\detokenize{\the\replacetoks}


\the\replacetoks

\edef\z{\the\replacetoks}

\detokenize{\edef\z{\the\replacetoks}\meaning\z}

\meaning\z
\nopagenumbers

\bye 
  • in the image in smaller print the #### are the way \detokenize prints ## from its input (which is seen at the bottom of the image, where now \meaning has been used but # at this stage has catcode 12). – user4686 Mar 14 '14 at 18:49
  • if anybody reads this, just to point out that unarmed is to be read unharmed. Knew something was fishy, took me a while to find out what. – user4686 Mar 14 '14 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.