# How do I put a “#” into a token list without getting two?

Consider this plain TeX file:

\newtoks\t
\t={#}
\showthe\t
\bye


If you compile it (with tex of pdftex), then in the output you get

> ##.
l.3 \showthe\t


Why do I get two of the #s?

(The background: I want to build the body of a macro definition in a token list, and the macro takes one argument that I want to put into the token list as #1. The idea is to use \edef\macro#1{\the\t}.)

-

When TeX shows #, it doubles it up. So in your

\newtoks\t
\t={#}
\showthe\t
\bye


example,

> ##.
l.3 \showthe\t


represents one # inside the token register. You can convince TeX to show this using \string:

\newtoks\t
\t={#}
\expandafter\string\the\t
\bye


(which of courses only works because there is a single token in \t!)

For the definition of the macro, you'd need something like

\newtoks\t
\t={You gave '#1'}
\expandafter\def\expandafter\test
\expandafter#\expandafter1\expandafter{\the\t}
\test{things}
\bye

-
Unfortunately, \detokenize{#} gives two ##. Is that a bug in pdfTeX? –  TH. Jan 4 '11 at 7:25
@TH. No, not a bug. \detokenize is meant to work very much like a toks, in the sense that you could do the same as \detokenize achieves by writing a toks to file and reading back with appropriate catcode changes. However, \detokenize is expandable whereas working via a file is not. If you try the file-based route, you'll also get two #. –  Joseph Wright Jan 4 '11 at 8:03
Yeah. I was actually hoping to use \detokenize to write a single # to the file which is where I ran across it. I forget the exact reason now. Still, I'd assume that a #_6 token would be replaced by a single #_12 token. E.g., I'd expect \scantokens\expandafter{\detokenize{\def\foo#1{#1}}} to work. –  TH. Jan 4 '11 at 8:21
Thanks for the explanation about \show. But why doesn't \edef\macro#1{\the\t} work? (I knew about the \expandafter workaround, but it's nasty, and I'm puzzled why this does work then.) –  Hendrik Vogt Jan 4 '11 at 8:55

Following what Yiannis wrote, you can do something like this:

% Put the replacement text into \t, using any computation like this doubling loop
\newtoks\t
\t{#1}
\count255 0
\loop\ifnum\count255<10
\t\expandafter\expandafter\expandafter{\expandafter\the\expandafter\t\the\t}
\repeat
% Define a helper that when executed will perform the real definition
\edef\helper{\noexpand\def\noexpand\X##1{\the\t}}
\helper
% Now the real macro \X can be used.
\X{[X] }
\bye

-
@TH.: That looks a bit clearer than the 5 \expandafter s in Joseph's suggestion, but I'm still puzzled why a simple \edef\macro#1{\the\t} doesn't work. (Fortunately, the initial reason for this post has gone since my \multexpafter got a lot simpler now.) –  Hendrik Vogt Jan 4 '11 at 17:25
@Hendrik This is designed as such in TeX. Tokens resulting from \the\token variable are not expanded further if the statement occurs inside an \edef macro definition. –  Yiannis Lazarides Jan 4 '11 at 19:07
@Yiannis: That's not the issue. The issue is that \t{#1}\edef\macro#1{\the\t} is the same as \def\macro#1{##1} which doesn't work: ! You can't use 'macro parameter character #' in horizontal mode. By stuffing it inside another macro, (\helper in my code) causes it to work. –  TH. Jan 4 '11 at 20:34
@TH.: Can you explain why \t{#1}\edef\macro#1{\the\t} is the same as \def\macro#1{##1}? I can't find this described in the TeXbook. –  Hendrik Vogt Jan 5 '11 at 17:24
@Hendrik: No. I cannot explain why that is the case. I can merely demonstrate that it is: \newtoks\t \t{#1} \edef\macro#1{\the\t} \show\macro \def\macro#1{##1} \show\macro \end. Best guess is it's a bug in TeX. I can't think of a reason that \ea\def\ea\macro\ea#\ea1\ea{\the\t} should behave differently from the \edef version. –  TH. Jan 5 '11 at 20:44

Joseph answered the reason why # shows up double, as the standard behaviour of show etc.

Since you want to build the body of a macro definition in a token list the only way to incorporate #1 etc. (IMHO) is to put the to put it via a macro.

\newtoks\t
\t={\def\z#1{\bf #1}}
\def\macro#1{\the\t \z{B} \z{#1}}
\macro{test}

\bye


The only difference from what was proposed by the OP, is that to put the body definition in a token list, you need to pass it to the macro by using \the\t and a \helpermacro{#1}.

-