11

If I make a command that takes one argument, and the argument should be allowed to take the unescaped default macro char: # as an argument and then treat it as a letter:

\documentclass{article}
\makeatletter
\def\allowhash#1{
  {\toks@{#1}\catcode`\#11\relax\scantokens\expandafter{\the\toks@}}%
}
\makeatother
\begin{document}
\allowhash{One hash: #, not two}
\end{document}

The above works except it prints the following: One hash: ##, not two. Is this expected behaviour? How could one fix this?

3
  • 1
    When the macro \allowhash is called, the category code of # is still 6, so it is doubled when the argument is absorbed. With a later \scantokens you get two # characters with category code 11 (12 would be a better choice). By the way, the group around the definition of \allowhash is useless, because you're changing no category code at definition time.
    – egreg
    Sep 2, 2018 at 11:27
  • @egreg aha! Removed the grouping (it was just remains from when I was experimenting). Sep 2, 2018 at 12:02

3 Answers 3

3

Let's look at your code:

\documentclass{article}
\makeatletter
\def\allowhash#1{
  {\toks@{#1}\catcode`\#11\relax\scantokens\expandafter{\the\toks@}}%
}
\makeatother
\begin{document}
\allowhash{One hash: #, not two}
\end{document}

LaTeX does fetch an argument for \allowhash, hereby tokenizing the tokens which form that argument under normal category-code-régime. Thus LaTeX does fetch an explicit-hash-character-token of category code 6(parameter) as a component of \allowhash's argument.

When expanding \allowhash, this token becomes a part of the content of \toks@.

Thus \toks@ contains an explicit-hash-character-token of category code 6(parameter).

Due to \expandafter...\the-trickery this explicit-hash-character-token of category code 6(parameter) ends up as a component of the ⟨general text⟩ of \scantokens.

\scantokens emulates unexpanded-writing the tokens from its ⟨general text⟩ into file and reading them back from the file and hereby tokenizing things under the current category-code-régime.

When writing to file or screen, explicit character tokens of category-code 6(parameter) get doubled.

Thus the hash gets doubled by \scantokens unexpanded-writing-part.

Off the cuff I can only offer a routine \ReplaceEveryHash which takes one argument and does replace each explicit catcode-6-character-token of the argument by its stringification—this mechanism does not act only on explicit catcode-6-hashes but on all explicit catcode-6-character-tokens.

\documentclass{article}

\makeatletter
%%=============================================================================
%% Paraphernalia:
%%    \UD@firstoftwo, \UD@secondoftwo,
%%    \UD@PassFirstToSecond, \UD@Exchange, \UD@removespace
%%    \UD@CheckWhetherNull, \UD@CheckWhetherBrace,
%%    \UD@CheckWhetherLeadingSpace, \UD@ExtractFirstArg
%%=============================================================================
\newcommand\UD@firstoftwo[2]{#1}%
\newcommand\UD@secondoftwo[2]{#2}%
\newcommand\UD@PassFirstToSecond[2]{#2{#1}}%
\newcommand\UD@Exchange[2]{#2#1}%
\newcommand\UD@removespace{}\UD@firstoftwo{\def\UD@removespace}{} {}%
%%-----------------------------------------------------------------------------
%% Check whether argument is empty:
%%.............................................................................
%% \UD@CheckWhetherNull{<Argument which is to be checked>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is empty>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is not empty>}%
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
%% <https://groups.google.com/forum/#!original/comp.text.tex/kuOEIQIrElc/lUg37FmhA74J>
\newcommand\UD@CheckWhetherNull[1]{%
  \romannumeral0\expandafter\UD@secondoftwo\string{\expandafter
  \UD@secondoftwo\expandafter{\expandafter{\string#1}\expandafter
  \UD@secondoftwo\string}\expandafter\UD@firstoftwo\expandafter{\expandafter
  \UD@secondoftwo\string}\expandafter\expandafter\UD@firstoftwo{ }{}%
  \UD@secondoftwo}{\expandafter\expandafter\UD@firstoftwo{ }{}\UD@firstoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether argument's first token is a catcode-1-character
%%.............................................................................
%% \UD@CheckWhetherBrace{<Argument which is to be checked>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked has leading
%%                        catcode-1-token>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked has no leading
%%                        catcode-1-token>}%
\newcommand\UD@CheckWhetherBrace[1]{%
  \romannumeral0\expandafter\UD@secondoftwo\expandafter{\expandafter{%
  \string#1.}\expandafter\UD@firstoftwo\expandafter{\expandafter
  \UD@secondoftwo\string}\expandafter\expandafter\UD@firstoftwo{ }{}%
  \UD@firstoftwo}{\expandafter\expandafter\UD@firstoftwo{ }{}\UD@secondoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether brace-balanced argument starts with a space-token
%%.............................................................................
%% \UD@CheckWhetherLeadingSpace{<Argument which is to be checked>}%
%%                             {<Tokens to be delivered in case <argument
%%                               which is to be checked>'s 1st token is a
%%                               space-token>}%
%%                             {<Tokens to be delivered in case <argument
%%                               which is to be checked>'s 1st token is not
%%                               a space-token>}%
\newcommand\UD@CheckWhetherLeadingSpace[1]{%
  \romannumeral0\UD@CheckWhetherNull{#1}%
  {\expandafter\expandafter\UD@firstoftwo{ }{}\UD@secondoftwo}%
  {\expandafter\UD@secondoftwo\string{\UD@CheckWhetherLeadingSpaceB.#1 }{}}%
}%
\newcommand\UD@CheckWhetherLeadingSpaceB{}%
\long\def\UD@CheckWhetherLeadingSpaceB#1 {%
  \expandafter\UD@CheckWhetherNull\expandafter{\UD@secondoftwo#1{}}%
  {\UD@Exchange{\UD@firstoftwo}}{\UD@Exchange{\UD@secondoftwo}}%
  {\UD@Exchange{ }{\expandafter\expandafter\expandafter\expandafter
   \expandafter\expandafter\expandafter}\expandafter\expandafter
   \expandafter}\expandafter\UD@secondoftwo\expandafter{\string}%
}%
%%-----------------------------------------------------------------------------
%% Extract first inner undelimited argument:
%%
%%   \UD@ExtractFirstArg{ABCDE} yields  {A}
%%
%%   \UD@ExtractFirstArg{{AB}CDE} yields  {AB}
%%.............................................................................
\newcommand\UD@RemoveTillUD@SelDOm{}%
\long\def\UD@RemoveTillUD@SelDOm#1#2\UD@SelDOm{{#1}}%
\newcommand\UD@ExtractFirstArg[1]{%
  \romannumeral0%
  \UD@ExtractFirstArgLoop{#1\UD@SelDOm}%
}%
\newcommand\UD@ExtractFirstArgLoop[1]{%
  \expandafter\UD@CheckWhetherNull\expandafter{\UD@firstoftwo{}#1}%
  { #1}%
  {\expandafter\UD@ExtractFirstArgLoop\expandafter{\UD@RemoveTillUD@SelDOm#1}}%
}%
%%=============================================================================
%% \ReplaceEveryHash{<argument>}%
%%
%%   Each explicit catcode-6(parameter)-character-token of the <argument> 
%%   will be replaced by its stringification.
%%
%%   You obtain the result after two expansion-steps, i.e., 
%%   in expansion-contexts you get the result after "hitting" 
%%   \ReplaceEveryHash by two \expandafter.
%%   
%%   As a side-effect, the routine does replace matching pairs of explicit
%%   character tokens of catcode 1 and 2 by matching pairs of curly braces
%%   of catcode 1 and 2.
%%   I suppose this won't be a problem in most situations as usually the
%%   curly braces are the only characters of category code 1 / 2...
%%
%%   This routine needs \detokenize from the eTeX extensions.
%%-----------------------------------------------------------------------------
\newcommand\ReplaceEveryHash[1]{%
   \romannumeral0\UD@ReplaceEveryHashLoop{#1}{}%
}%
\newcommand\UD@ReplaceEveryHashLoop[2]{%
  \UD@CheckWhetherNull{#1}{ #2}{%
    \UD@CheckWhetherLeadingSpace{#1}{%
       \expandafter\UD@ReplaceEveryHashLoop
       \expandafter{\UD@removespace#1}{#2 }%
    }{%
      \UD@CheckWhetherBrace{#1}{%
        \expandafter\expandafter\expandafter\UD@PassFirstToSecond
        \expandafter\expandafter\expandafter{%
        \expandafter\UD@PassFirstToSecond\expandafter{%
            \romannumeral0\expandafter\UD@ReplaceEveryHashLoop
            \romannumeral0\UD@ExtractFirstArgLoop{#1\UD@SelDOm}{}%
        }{#2}}%
        {\expandafter\UD@ReplaceEveryHashLoop
         \expandafter{\UD@firstoftwo{}#1}}%
      }{%
       \expandafter\UD@CheckWhetherHash
       \romannumeral0\UD@ExtractFirstArgLoop{#1\UD@SelDOm}{#1}{#2}%
      }%
    }%
  }%
}%
\newcommand\UD@CheckWhetherHash[3]{%
  \expandafter\UD@CheckWhetherLeadingSpace\expandafter{\string#1}{%
    \expandafter\expandafter\expandafter\UD@CheckWhetherNull
    \expandafter\expandafter\expandafter{%
    \expandafter\UD@removespace\string#1}{%
      \expandafter\expandafter\expandafter\UD@CheckWhetherNull
      \expandafter\expandafter\expandafter{%
      \expandafter\UD@removespace\detokenize{#1}}{%
        \expandafter\UD@ReplaceEveryHashLoop
        \expandafter{\UD@firstoftwo{}#2}{#3#1}%
      }{%
        \expandafter\expandafter\expandafter\UD@PassFirstToSecond
        \expandafter\expandafter\expandafter{\expandafter\UD@Exchange
        \expandafter{\string#1}{#3}}{%
          \expandafter\UD@ReplaceEveryHashLoop
          \expandafter{\UD@firstoftwo{}#2}%
        }%
      }%
    }{%
      \expandafter\UD@ReplaceEveryHashLoop
      \expandafter{\UD@firstoftwo{}#2}{#3#1}%
    }%
  }{%
    \expandafter\expandafter\expandafter\UD@CheckWhetherNull
    \expandafter\expandafter\expandafter{%
    \expandafter\UD@firstoftwo
    \expandafter{\expandafter}\string#1}{%
      \expandafter\expandafter\expandafter\UD@CheckWhetherNull
      \expandafter\expandafter\expandafter{%
      \expandafter\UD@firstoftwo
      \expandafter{\expandafter}\detokenize{#1}}{%
        \expandafter\UD@ReplaceEveryHashLoop
        \expandafter{\UD@firstoftwo{}#2}{#3#1}%
      }{%
        \expandafter\expandafter\expandafter\UD@PassFirstToSecond
        \expandafter\expandafter\expandafter{\expandafter\UD@Exchange
        \expandafter{\string#1}{#3}}{%
          \expandafter\UD@ReplaceEveryHashLoop
          \expandafter{\UD@firstoftwo{}#2}%
        }%
      }%
    }{%
      \expandafter\UD@ReplaceEveryHashLoop
      \expandafter{\UD@firstoftwo{}#2}{#3#1}%
    }%
  }%
}%
%----------------------------------------------------------------------

\newcommand\allowhash[1]{\ReplaceEveryHash{#1}}

\newcommand\allowanddetokenizehash[1]{%
  \detokenize\expandafter\expandafter\expandafter{\ReplaceEveryHash{#1}}%
}%

\makeatother

\begin{document}
\allowhash{One hash: #, not two}

\begingroup
\frenchspacing
\ttfamily \allowanddetokenizehash{One hash: #, not two. This time in braces:{#{#{#{#}}}#}}
\endgroup

For comparison the effect of \verb|\detokenize| without prior hash-replacing:

\begingroup
\frenchspacing
\ttfamily \detokenize{One hash: #, not two. This time in braces:{#{#{#{#}}}#}}
\endgroup

\end{document}

enter image description here

11

enter image description here

\documentclass{article}
\makeatletter
\def\allowhash{%%%%
    \bgroup\everyeof{\egroup}\catcode35=11\relax\scantokens}%
\makeatother
\begin{document}
\allowhash{One hash: #, not two}

\end{document}

or in a macro as requested in comments

enter image description here

\documentclass{article}
\makeatletter
\def\allowhash{%%%%
    \bgroup\catcode35=11\relax\afterassignment\egroup\gdef\foo}%
\makeatother
\begin{document}
\allowhash{One hash: #, not two}

[\foo]  [\foo]
\end{document}
4
  • What if I wanted the result in a macro instead of just printing it? Sep 2, 2018 at 11:35
  • @AndreasStorvikStrauman added a macro version Sep 2, 2018 at 11:39
  • Could you add some words of explanation how the first macro works?
    – siracusa
    Sep 2, 2018 at 15:00
  • @siracusa it just makes # non-special without parsing the {...} as an argument of \allowhash, the \scantokens isn't really needed at all except the end of file hook is used to insert an endgroup to restore the meaning of # Sep 2, 2018 at 15:13
6

When \scantokens acts, the # characters are already doubled, because they're absorbed as the argument to a macro.

With the l3regex module of expl3 it's easier:

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn

\NewDocumentCommand{\allowhash}{m}
 {
  \tl_set:Nn \l_tmpa_tl { #1 }
  \regex_replace_all:nnN { \cP\# } { \cO\# } \l_tmpa_tl
  \tl_use:N \l_tmpa_tl
 }
\ExplSyntaxOff

\begin{document}

\allowhash{One hash: #, not two}

\end{document}

enter image description here

You can easily add support for saving the token list in a macro.

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn

\NewDocumentCommand{\allowhash}{om}
 {
  \tl_set:Nn \l_tmpa_tl { #2 }
  \regex_replace_all:nnN { \cP\# } { \cO\# } \l_tmpa_tl
  \IfNoValueTF { #1 }
   { \tl_use:N \l_tmpa_tl }
   { \tl_set_eq:NN #1 \l_tmpa_tl }
 }

\ExplSyntaxOff

\begin{document}

\allowhash{One hash: #, not two}

\allowhash[\foo]{One hash: #, not two}

\texttt{\meaning\foo}

\end{document}

enter image description here

(There seems to be a double space in the picture, but it's only due to \nonfrenchspacing.)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .