Macro char # doubles when made letter?

If I make a command that takes one argument, and the argument should be allowed to take the unescaped default macro char: # as an argument and then treat it as a letter:

\documentclass{article}
\makeatletter
\def\allowhash#1{
{\toks@{#1}\catcode\#11\relax\scantokens\expandafter{\the\toks@}}%
}
\makeatother
\begin{document}
\allowhash{One hash: #, not two}
\end{document}


The above works except it prints the following: One hash: ##, not two. Is this expected behaviour? How could one fix this?

• When the macro \allowhash is called, the category code of # is still 6, so it is doubled when the argument is absorbed. With a later \scantokens you get two # characters with category code 11 (12 would be a better choice). By the way, the group around the definition of \allowhash is useless, because you're changing no category code at definition time. Sep 2, 2018 at 11:27
• @egreg aha! Removed the grouping (it was just remains from when I was experimenting). Sep 2, 2018 at 12:02
• Apr 21 at 5:59

\documentclass{article}
\makeatletter
\def\allowhash#1{
{\toks@{#1}\catcode\#11\relax\scantokens\expandafter{\the\toks@}}%
}
\makeatother
\begin{document}
\allowhash{One hash: #, not two}
\end{document}


LaTeX does fetch an argument for \allowhash, hereby tokenizing the tokens which form that argument under normal category-code-régime. Thus LaTeX does fetch an explicit-hash-character-token of category code 6(parameter) as a component of \allowhash's argument.

When expanding \allowhash, this token becomes a part of the content of \toks@.

Thus \toks@ contains an explicit-hash-character-token of category code 6(parameter).

Due to \expandafter...\the-trickery this explicit-hash-character-token of category code 6(parameter) ends up as a component of the ⟨general text⟩ of \scantokens.

\scantokens emulates unexpanded-writing the tokens from its ⟨general text⟩ into file and reading them back from the file and hereby tokenizing things under the current category-code-régime.

When writing to file or screen, explicit character tokens of category-code 6(parameter) get doubled.

Thus the hash gets doubled by \scantokens unexpanded-writing-part.

Off the cuff I can only offer a routine \ReplaceEveryHash which takes one argument and does replace each explicit catcode-6-character-token of the argument by its stringification—this mechanism does not act only on explicit catcode-6-hashes but on all explicit catcode-6-character-tokens.

\documentclass{article}

\makeatletter
%%=============================================================================
%% Paraphernalia:
%%    \UD@firstoftwo, \UD@secondoftwo,
%%    \UD@PassFirstToSecond, \UD@Exchange, \UD@removespace
%%    \UD@CheckWhetherNull, \UD@CheckWhetherBrace,
%%=============================================================================
\newcommand\UD@firstoftwo[2]{#1}%
\newcommand\UD@secondoftwo[2]{#2}%
\newcommand\UD@PassFirstToSecond[2]{#2{#1}}%
\newcommand\UD@Exchange[2]{#2#1}%
\newcommand\UD@removespace{}\UD@firstoftwo{\def\UD@removespace}{} {}%
%%-----------------------------------------------------------------------------
%% Check whether argument is empty:
%%.............................................................................
%% \UD@CheckWhetherNull{<Argument which is to be checked>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is empty>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is not empty>}%
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
\newcommand\UD@CheckWhetherNull[1]{%
\romannumeral0\expandafter\UD@secondoftwo\string{\expandafter
\UD@secondoftwo\expandafter{\expandafter{\string#1}\expandafter
\UD@secondoftwo\string}\expandafter\UD@firstoftwo\expandafter{\expandafter
\UD@secondoftwo\string}\expandafter\expandafter\UD@firstoftwo{ }{}%
\UD@secondoftwo}{\expandafter\expandafter\UD@firstoftwo{ }{}\UD@firstoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether argument's first token is a catcode-1-character
%%.............................................................................
%% \UD@CheckWhetherBrace{<Argument which is to be checked>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked has leading
%%                        catcode-1-token>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked has no leading
%%                        catcode-1-token>}%
\newcommand\UD@CheckWhetherBrace[1]{%
\romannumeral0\expandafter\UD@secondoftwo\expandafter{\expandafter{%
\string#1.}\expandafter\UD@firstoftwo\expandafter{\expandafter
\UD@secondoftwo\string}\expandafter\expandafter\UD@firstoftwo{ }{}%
\UD@firstoftwo}{\expandafter\expandafter\UD@firstoftwo{ }{}\UD@secondoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether brace-balanced argument starts with a space-token
%%.............................................................................
%% \UD@CheckWhetherLeadingSpace{<Argument which is to be checked>}%
%%                             {<Tokens to be delivered in case <argument
%%                               which is to be checked>'s 1st token is a
%%                               space-token>}%
%%                             {<Tokens to be delivered in case <argument
%%                               which is to be checked>'s 1st token is not
%%                               a space-token>}%
\romannumeral0\UD@CheckWhetherNull{#1}%
{\expandafter\expandafter\UD@firstoftwo{ }{}\UD@secondoftwo}%
}%
\expandafter\UD@CheckWhetherNull\expandafter{\UD@secondoftwo#1{}}%
{\UD@Exchange{\UD@firstoftwo}}{\UD@Exchange{\UD@secondoftwo}}%
{\UD@Exchange{ }{\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter}\expandafter\expandafter
\expandafter}\expandafter\UD@secondoftwo\expandafter{\string}%
}%
%%-----------------------------------------------------------------------------
%% Extract first inner undelimited argument:
%%
%%   \UD@ExtractFirstArg{ABCDE} yields  {A}
%%
%%   \UD@ExtractFirstArg{{AB}CDE} yields  {AB}
%%.............................................................................
\newcommand\UD@RemoveTillUD@SelDOm{}%
\long\def\UD@RemoveTillUD@SelDOm#1#2\UD@SelDOm{{#1}}%
\newcommand\UD@ExtractFirstArg[1]{%
\romannumeral0%
\UD@ExtractFirstArgLoop{#1\UD@SelDOm}%
}%
\newcommand\UD@ExtractFirstArgLoop[1]{%
\expandafter\UD@CheckWhetherNull\expandafter{\UD@firstoftwo{}#1}%
{ #1}%
{\expandafter\UD@ExtractFirstArgLoop\expandafter{\UD@RemoveTillUD@SelDOm#1}}%
}%
%%=============================================================================
%% \ReplaceEveryHash{<argument>}%
%%
%%   Each explicit catcode-6(parameter)-character-token of the <argument>
%%   will be replaced by its stringification.
%%
%%   You obtain the result after two expansion-steps, i.e.,
%%   in expansion-contexts you get the result after "hitting"
%%   \ReplaceEveryHash by two \expandafter.
%%
%%   As a side-effect, the routine does replace matching pairs of explicit
%%   character tokens of catcode 1 and 2 by matching pairs of curly braces
%%   of catcode 1 and 2.
%%   I suppose this won't be a problem in most situations as usually the
%%   curly braces are the only characters of category code 1 / 2...
%%
%%   This routine needs \detokenize from the eTeX extensions.
%%-----------------------------------------------------------------------------
\newcommand\ReplaceEveryHash[1]{%
\romannumeral0\UD@ReplaceEveryHashLoop{#1}{}%
}%
\newcommand\UD@ReplaceEveryHashLoop[2]{%
\UD@CheckWhetherNull{#1}{ #2}{%
\expandafter\UD@ReplaceEveryHashLoop
\expandafter{\UD@removespace#1}{#2 }%
}{%
\UD@CheckWhetherBrace{#1}{%
\expandafter\expandafter\expandafter\UD@PassFirstToSecond
\expandafter\expandafter\expandafter{%
\expandafter\UD@PassFirstToSecond\expandafter{%
\romannumeral0\expandafter\UD@ReplaceEveryHashLoop
\romannumeral0\UD@ExtractFirstArgLoop{#1\UD@SelDOm}{}%
}{#2}}%
{\expandafter\UD@ReplaceEveryHashLoop
\expandafter{\UD@firstoftwo{}#1}}%
}{%
\expandafter\UD@CheckWhetherHash
\romannumeral0\UD@ExtractFirstArgLoop{#1\UD@SelDOm}{#1}{#2}%
}%
}%
}%
}%
\newcommand\UD@CheckWhetherHash[3]{%
\expandafter\expandafter\expandafter\UD@CheckWhetherNull
\expandafter\expandafter\expandafter{%
\expandafter\UD@removespace\string#1}{%
\expandafter\expandafter\expandafter\UD@CheckWhetherNull
\expandafter\expandafter\expandafter{%
\expandafter\UD@removespace\detokenize{#1}}{%
\expandafter\UD@ReplaceEveryHashLoop
\expandafter{\UD@firstoftwo{}#2}{#3#1}%
}{%
\expandafter\expandafter\expandafter\UD@PassFirstToSecond
\expandafter\expandafter\expandafter{\expandafter\UD@Exchange
\expandafter{\string#1}{#3}}{%
\expandafter\UD@ReplaceEveryHashLoop
\expandafter{\UD@firstoftwo{}#2}%
}%
}%
}{%
\expandafter\UD@ReplaceEveryHashLoop
\expandafter{\UD@firstoftwo{}#2}{#3#1}%
}%
}{%
\expandafter\expandafter\expandafter\UD@CheckWhetherNull
\expandafter\expandafter\expandafter{%
\expandafter\UD@firstoftwo
\expandafter{\expandafter}\string#1}{%
\expandafter\expandafter\expandafter\UD@CheckWhetherNull
\expandafter\expandafter\expandafter{%
\expandafter\UD@firstoftwo
\expandafter{\expandafter}\detokenize{#1}}{%
\expandafter\UD@ReplaceEveryHashLoop
\expandafter{\UD@firstoftwo{}#2}{#3#1}%
}{%
\expandafter\expandafter\expandafter\UD@PassFirstToSecond
\expandafter\expandafter\expandafter{\expandafter\UD@Exchange
\expandafter{\string#1}{#3}}{%
\expandafter\UD@ReplaceEveryHashLoop
\expandafter{\UD@firstoftwo{}#2}%
}%
}%
}{%
\expandafter\UD@ReplaceEveryHashLoop
\expandafter{\UD@firstoftwo{}#2}{#3#1}%
}%
}%
}%
%----------------------------------------------------------------------

\newcommand\allowhash[1]{\ReplaceEveryHash{#1}}

\newcommand\allowanddetokenizehash[1]{%
\detokenize\expandafter\expandafter\expandafter{\ReplaceEveryHash{#1}}%
}%

\makeatother

\begin{document}
\allowhash{One hash: #, not two}

\begingroup
\frenchspacing
\ttfamily \allowanddetokenizehash{One hash: #, not two. This time in braces:{#{#{#{#}}}#}}
\endgroup

For comparison the effect of \verb|\detokenize| without prior hash-replacing:

\begingroup
\frenchspacing
\ttfamily \detokenize{One hash: #, not two. This time in braces:{#{#{#{#}}}#}}
\endgroup

\end{document}


\documentclass{article}
\makeatletter
\def\allowhash{%%%%
\bgroup\everyeof{\egroup}\catcode35=11\relax\scantokens}%
\makeatother
\begin{document}
\allowhash{One hash: #, not two}

\end{document}


or in a macro as requested in comments

\documentclass{article}
\makeatletter
\def\allowhash{%%%%
\bgroup\catcode35=11\relax\afterassignment\egroup\gdef\foo}%
\makeatother
\begin{document}
\allowhash{One hash: #, not two}

[\foo]  [\foo]
\end{document}

• What if I wanted the result in a macro instead of just printing it? Sep 2, 2018 at 11:35
• @AndreasStorvikStrauman added a macro version Sep 2, 2018 at 11:39
• Could you add some words of explanation how the first macro works? Sep 2, 2018 at 15:00
• @siracusa it just makes # non-special without parsing the {...} as an argument of \allowhash, the \scantokens isn't really needed at all except the end of file hook is used to insert an endgroup to restore the meaning of # Sep 2, 2018 at 15:13

When \scantokens acts, the # characters are already doubled, because they're absorbed as the argument to a macro.

With the l3regex module of expl3 it's easier:

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn

\NewDocumentCommand{\allowhash}{m}
{
\tl_set:Nn \l_tmpa_tl { #1 }
\regex_replace_all:nnN { \cP\# } { \cO\# } \l_tmpa_tl
\tl_use:N \l_tmpa_tl
}
\ExplSyntaxOff

\begin{document}

\allowhash{One hash: #, not two}

\end{document}


You can easily add support for saving the token list in a macro.

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn

\NewDocumentCommand{\allowhash}{om}
{
\tl_set:Nn \l_tmpa_tl { #2 }
\regex_replace_all:nnN { \cP\# } { \cO\# } \l_tmpa_tl
\IfNoValueTF { #1 }
{ \tl_use:N \l_tmpa_tl }
{ \tl_set_eq:NN #1 \l_tmpa_tl }
}

\ExplSyntaxOff

\begin{document}

\allowhash{One hash: #, not two}

\allowhash[\foo]{One hash: #, not two}

\texttt{\meaning\foo}

\end{document}


(There seems to be a double space in the picture, but it's only due to \nonfrenchspacing.)