# Appending a character to a token list using its charcode

Is there any way to obtain the actual character token from its character code representation? In particular, I would like to have a macro \prepend#1#2 that takes a token list #1 and prepends the character token corresponding to the character code #2 to it.

Here is a short demonstration of the desired outcome:

\newtoks\test
\test={bc}
\prepend\test{97}
\showthe\test % should print abc

• You can use expl3's \char_generate:nn, but it doesn't work as you want: the characters it uses are from token lists like \c__char_127_tl. Even then, it cannot generate an invalid character token. You might be able to generate an invalid char with LuaTeX, at best – Phelype Oleinik Mar 23 '20 at 23:25
• I'm not sure what you mean: \char127 just prints the character in slot 127 of the current font and has nothing to do with the character token with code 127. Anyway, no character with category code “invalid” can ever go beyond TeX's eyes, so it cannot appear in a token list. – egreg Mar 23 '20 at 23:35
• @PhelypeOleinik @egreg the invalid char was a bad example (and the use of \char was incorrect), I have updated my question. I need to generate a character token with character code #2, the category code does not matter. – wave Mar 23 '20 at 23:42
• But how do you know that 97 should be a? \char is font-dependent, eg. in cmsy10 \char97 yields ⊣ – Robert Mar 24 '20 at 0:02
• @Robert I meant the character code (not what follows \char), i.e. the number TeX associates with every character of input – wave Mar 24 '20 at 0:09

You can use \char_generate:nn { <charcode> } { <catcode> }:

\input expl3-generic
\ExplSyntaxOn
\cs_new_eq:NN \toks_use:N \tex_the:D
\cs_new_protected:Npn \prepend #1 #2
{
\if:w \exp_not:N #1 #1
\use:x { #1 = { \char_generate:nn {#2} { 12 } \toks_use:N #1 } }
\else:
\tl_put_left:Nx #1 { \char_generate:nn {#2} { 12 } }
\fi:
}
\ExplSyntaxOff

\newtoks\test
\test={bc}
\prepend\test{97}
\showthe\test % should print abc

\def\test{bc}
\prepend\test{97}
\show\test % should print abc

\bye


The terminal output will be:

> abc.
l.16 \showthe\test
% should print abc
?
> \test=macro:
->abc.
l.21 \show\test
% should print abc again
?


How \char_generate:nn generates the characters depends on which engine is in use. In LuaTeX it uses tex.cprint(<catcode>, utf8_char(<charcode>)), in a similar way to Henri's answer, but with a possible <catcode> setting. In XeTeX it uses \Ucharcat <charcode> <catcode>.

In the other engines supported by expl3 (pdftex, ε-pTeX, and ε-upTeX) there is no way to actually generate characters in expansion-only contexts (a key feature of \char_generate:nn), so expl3 pre-generates these characters using the same approach as in egreg's answer, and then \char_generate:nn just uses the characters when asked for.

As in egreg's answer, you can't generate characters of some catcodes—namely 0, 5, 9, 14, and 15—because they don't produce tokens (they disappear when TeX is scanning the input, so they don't exist at the macro-expansion level). Also, the expl3 implementation does not allow the generation of space characters for consistency across engines, because the Lua version doesn't allow so. However since you want a Knuth TeX version, space chars are also allowed.

The code below is an adaptation of the expl3 code for \char_generate:nn modified to work in Knuth TeX. The code is basically the same, except that a few more complications are necessary mainly due to the lack of \unexpanded, which allows you to have single parameter tokens in a macro, and allows you to easily append stuff to a macro without the need of a toks register. Other than that, it's the same thing.

The code first defines a temporary toks register which contains the null character (^^@) with the different possible catcodes, separated by \or:

\or ^^@% 1
\or ^^@% 2
\or ^^@% 3
\or ^^@% 4
\or    % 5 Invalid
\or ^^@^^@% 6 Has to be doubled for a later \def
\or ^^@% 7
\or ^^@% 8
\or    % 9 Invalid
\or ^^@% 10
\or ^^@% 11
\or ^^@% 12
\or ^^@% 13


then it loops through all 256 character codes and sets the \lccode of the null character to #1 and then uses the \lowercase trick in egreg's answer:

    \begingroup
\lccode0=#1
\lccode32=#1
\edef\x{\endgroup
\gdef\expandafter\noexpand
\csname c__char_\romannumeral#1_tl\endcsname{\the\tmptoks}}%
\lowercase\expandafter{\x}


which for a character code, say 97, results in:

\gdef\c__char_xcvii_tl{\or a\or a\or a\or a\or \or aa\or a\or a\or \or a\or a\or a\or a}


then given a character code <charcode> you can access that token list with \csname c__char_\romannumeral<charcode>_tl\endcsname, and then with \ifcase\fi you have the requested character.

The \chargenerate macro first checks (in \generateaux) if the arguments are in a valid range (catcode between 1 and 13, except 5 and 9, and charcode between 0 and 255, though with Knuth TeX you might need to change that to 127), and then calls \generateauxi with the arguments, which then uses the \ifcase test above (with a few more bits and pieces for expansion control) to leave the requested character.

Running the code below with tex I get:

% Auxiliaries
\long\def\gobbletoqstop#1\qstop{}
\long\def\firstofone#1{#1}
\chardef\expend=0
% Expandable error message
\begingroup
\long\xdef\expandableerror#1{%
\noexpand\expandafter\noexpand\expandafter\noexpand\expandafter
\noexpand\gobbletoqstop\noexpand\firstofone
{\csname Error! \endcsname#1}\noexpand\qstop}
\endgroup
% Append stuff to a toks register
\def\toksputright#1{%
\begingroup
\def\toksputtoks{#1}%
\afterassignment\toksputrightaux
\toks0=}
\def\toksputrightaux{%
\edef\x{\endgroup
\toksputtoks={\the\toksputtoks\the\toks0}}%
\x}
% Set up constant token lists
\newtoks\tmptoks
\begingroup
\tmptoks{ \noexpand\or}%
\catcode0=1
\toksputright\tmptoks{^^@\iffalse}}%
\catcode0=2
\toksputright\tmptoks{{\fi\noexpand\or^^@}%
\begingroup
\def\noop{}%
\edef\x{\expandafter\noop\the\tmptoks}%
\expandafter\endgroup
\expandafter\tmptoks\expandafter{\x}%
\catcode0=3  \toksputright\tmptoks{\or^^@}%
\catcode0=4  \toksputright\tmptoks{\or^^@}%
\catcode0=5  \toksputright\tmptoks{\or}%
\catcode0=6  \toksputright\tmptoks{\or^^@^^@}%
\catcode0=7  \toksputright\tmptoks{\or^^@}%
\catcode0=8  \toksputright\tmptoks{\or^^@}%
\catcode0=9  \toksputright\tmptoks{\or}%
\catcode0=10 \toksputright\tmptoks\expandafter{\firstofone{\or}^^@}%
\catcode0=11 \toksputright\tmptoks{\or ^^@}%
\catcode0=12 \toksputright\tmptoks{\or^^@}%
\catcode0=13 \toksputright\tmptoks{\or^^@}%
\def\chartmp#1;{%
\begingroup
\lccode0=#1
\lccode32=#1
\edef\x{\endgroup
\gdef\expandafter\noexpand
\csname c__chargen_\romannumeral#1_tl\endcsname{\the\tmptoks}}%
\lowercase\expandafter{\x}}%
\let^^L\relax
\catcode^^L=12
\count0=0
\loop
\expandafter\chartmp\number\count0;
\ifnum\count0<256 \repeat
\endgroup
% Main definition
\def\chargenerate#1#2{%
\romannumeral\expandafter\generateaux
\number#1\expandafter;\number#2;}
% Check for invalid input
\def\generateaux#1;#2;{%
\ifnum0%
\ifnum#1=0  1\fi
\ifnum#2=10 1\fi
=11
\expandableerror{Cannot generate null char as a space.}%
\else
\ifodd0%
\ifnum#2< 1 1\fi
\ifnum#2= 5 1\fi
\ifnum#2= 9 1\fi
\ifnum#2>13 1\fi\space
\expandableerror{Invalid catcode for char generation.}%
\else
\ifodd0%
\ifnum#1<  0 1\fi
\ifnum#1>"FF 1\fi\space
\expandableerror{Charcode requested out of engine range.}%
\else
\generateauxi{#1}{#2}%
\fi
\fi
\fi
\expend}
% Actual char generation
\def\generateauxi#1#2#3\expend{%
#3%
\iffalse{\fi
\expandafter\expandafter
\expandafter\expend
\expandafter\expandafter
\ifcase#2%
\csname c__chargen_\romannumeral#1_tl\endcsname
\or}
\fi}

% Testing
\def\empty{}
\begingroup
\lccode\~=a
\lowercase{\endgroup
\gdef ~{\ active character a}%
}
\def\test#1{%
\edef\x{%
\ifnum#1=2 {\iffalse}\fi\space\noexpand\meaning\fi % add { if a is a }
\chargenerate{97}{#1}%
\ifnum#1=6 \chargenerate{97}{#1}\fi% add another # if a is a #
\ifnum#1=1 \iffalse{\fi\space\noexpand\meaning}\fi % if a is a {, add a }
}%
\ifx\x\empty
#1: ERROR
\else
#1: \expandafter\meaning\x
\fi\par}

\tt\scrollmode
\count2=0
\loop
\test{\the\count2 }%
\ifnum\count2<16
\repeat

\bye

• thank you. as I understand, expl3 works with e-TeX, right? out of curiosity, do you know if the solution actually relies on e-TeX extensions (i.e. would it be possible to do it without them)? – wave Mar 24 '20 at 0:01
• @wave Yes, expl3 requires e-TeX. The implementation of \char_generate:nn is different depending on the engine: for LuaTeX it uses cprint(catcode, utf8_char(charcode)), for XeTeX it uses \Ucharcat, and for the other engines it's a token list-based approach (only 256 characters are available) which stores each charcode with different catcodes in macros, which are set up more or less as in egreg's answer, and then can be queried expandably. The latter doesn't require e-TeX as far as I can see (I didn't dig too deep in the code, but if you want I can make a non-expl3 version tomorrow). – Phelype Oleinik Mar 24 '20 at 0:46
• well I would find it interesting to see in more detail how the "other engine" case is handled. I just tried to look into the source code but since I unfortunately don't know any expl3 I couldn't figure anything out ... – wave Mar 24 '20 at 9:26
• I understand it now, that's quite brilliant (to me at least :D). your answer addresses not only my question, but also all the questions I should have asked; awesome. – wave Mar 24 '20 at 19:06
• @wave You might want to see the edit :-) – Phelype Oleinik Mar 26 '20 at 14:18

\lowercase is a good way to do this, with any TeX.

\def\prepend#1#2{% toks, charcode
\begingroup
\lccode9=#2\relax
\lowercase{%
\edef\0{\endgroup
#1={9\the#1}}%
\0}}


Assumes the toks register is not \0.

\newtoks\test

\def\prepend#1#2{%
\ifcase\catcode#2\relax
% 0, do nothing
\or
% 1, do nothing
\or
% 2, do nothing
\or
\prependaux#1{#2}{$}% 3 \or \prependaux#1{#2}{&}% 4 \or % 5, do nothing \or \prependaux#1{#2}{##}% 6 \or \prependaux#1{#2}{^}% 7 \or \prependaux#1{#2}{_}% 8 \or % 9, do nothing \or \prependaux#1{#2}{ }% 10 \or \prependaux#1{#2}{a}% 11 \or \prependaux#1{#2}{?}% 12 \or \prependaux#1{#2}{~}% 13 % 14 or 15, do nothing \fi } \def\prependaux#1#2#3{% \begingroup\lccode#3=#2\relax \lowercase{\endgroup\toks0={#3}}% #1\expandafter{\the\toks\expandafter0\the#1}% } \test={bc} \prepend\test{97} \message{\number?} \catcode?=3 \prepend\test{63} \the\test$

\prepend\test{\#}

\showthe\test

\bye


You can't add characters with category codes 0, 1, 2, 5, 9, 14 or 15.

As you see, I prepended a “strange” category code 3 character and the code \the\test\$ prints a math formula.

Restriction: #1 cannot be \toks0.

• that's a pretty good idea. as I don't really care about the category code, the lowercase trick alone is even enough! – wave Mar 24 '20 at 0:18
• @wave Why don't go the whole road? ;-) – egreg Mar 24 '20 at 0:25
• well I do want to have those category 0,1,2 etc. chars in the token list :) I'll probably just add a cat 12 char as the third argument to \prependaux in those remaining cases – wave Mar 24 '20 at 0:32

You can use LuaTeX and the string.char function to convert an ASCII code into the corresponding character.

\newtoks\test
\test={bc}
\tokspre\test\expandafter{\directlua{tex.sprint(string.char(97))}}
\showthe\test
\bye

• Wow this is cool; I'm surprised this works! Is it because .. calls some custom concatenation code, or is LuaTeX's internal representation of a TeX token list just a Lua string? Probably the former I guess… – ShreevatsaR Mar 24 '20 at 1:12
• Okay, the syntax looks a little bit like magic, but it's essentially just operator overloading for this equivalent code: tex.settoks("test", string.char(97) .. tex.gettoks("test")). The tex.gettoks just converts the TeX tokens to a string. This also means that there is actually a big caveat here, because tex.settoks actually retokenizes the input string, setting all catcodes to 12 (except space which is still 10). – Henri Menke Mar 24 '20 at 2:24
• But I think you should also be able to do \tokspre\test\expandafter{\directlua{tex.sprint(string.char(97))}}` which doesn't have the retokenization problem. – Henri Menke Mar 24 '20 at 2:26
• I've actually decided to replace it, because the retokenization is something you usually don't want. – Henri Menke Mar 24 '20 at 2:33
• That's for my mistake. – Sebastiano Mar 24 '20 at 23:38