8

If I make a command that takes one argument, and the argument should be allowed to take the unescaped default macro char: # as an argument and then treat it as a letter:

\documentclass{article}
\makeatletter
\def\allowhash#1{
  {\toks@{#1}\catcode`\#11\relax\scantokens\expandafter{\the\toks@}}%
}
\makeatother
\begin{document}
\allowhash{One hash: #, not two}
\end{document}

The above works except it prints the following: One hash: ##, not two. Is this expected behaviour? How could one fix this?

  • 1
    When the macro \allowhash is called, the category code of # is still 6, so it is doubled when the argument is absorbed. With a later \scantokens you get two # characters with category code 11 (12 would be a better choice). By the way, the group around the definition of \allowhash is useless, because you're changing no category code at definition time. – egreg Sep 2 '18 at 11:27
  • @egreg aha! Removed the grouping (it was just remains from when I was experimenting). – Andreas Storvik Strauman Sep 2 '18 at 12:02
2

Let's look at your code:

\documentclass{article}
\makeatletter
\def\allowhash#1{
  {\toks@{#1}\catcode`\#11\relax\scantokens\expandafter{\the\toks@}}%
}
\makeatother
\begin{document}
\allowhash{One hash: #, not two}
\end{document}

LaTeX does fetch an argument for \allowhash, hereby tokenizing the tokens which form that argument under normal category-code-régime. Thus LaTeX does fetch an explicit-hash-character-token of category code 6(parameter) as a component of \allowhash's argument.

When expanding \allowhash, this token becomes a part of the content of \toks@.

Thus \toks@ contains an explicit-hash-character-token of category code 6(parameter).

Due to \expandafter...\the-trickery this explicit-hash-character-token of category code 6(parameter) ends up as a component of the ⟨general text⟩ of \scantokens.

\scantokens emulates unexpanded-writing the tokens from its ⟨general text⟩ into file and reading them back from the file and hereby tokenizing things under the current category-code-régime.

When writing to file or screen, explicit character tokens of category-code 6(parameter) get doubled.

Thus the hash gets doubled by \scantokens unexpanded-writing-part.

Off the cuff I can only offer a routine \ReplaceEveryHash which takes one argument and does replace each explicit catcode-6-character-token of the argument by its stringification—this mechanism does not act only on explicit catcode-6-hashes but on all explicit catcode-6-character-tokens.

\documentclass{article}

\makeatletter
%%=============================================================================
%% Paraphernalia:
%%    \UD@firstoftwo, \UD@secondoftwo,
%%    \UD@PassFirstToSecond, \UD@Exchange, \UD@removespace
%%    \UD@CheckWhetherNull, \UD@CheckWhetherBrace,
%%    \UD@CheckWhetherLeadingSpace, \UD@ExtractFirstArg
%%=============================================================================
\newcommand\UD@firstoftwo[2]{#1}%
\newcommand\UD@secondoftwo[2]{#2}%
\newcommand\UD@PassFirstToSecond[2]{#2{#1}}%
\newcommand\UD@Exchange[2]{#2#1}%
\newcommand\UD@removespace{}\UD@firstoftwo{\def\UD@removespace}{} {}%
%%-----------------------------------------------------------------------------
%% Check whether argument is empty:
%%.............................................................................
%% \UD@CheckWhetherNull{<Argument which is to be checked>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is empty>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is not empty>}%
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
%% <https://groups.google.com/forum/#!original/comp.text.tex/kuOEIQIrElc/lUg37FmhA74J>
\newcommand\UD@CheckWhetherNull[1]{%
  \romannumeral0\expandafter\UD@secondoftwo\string{\expandafter
  \UD@secondoftwo\expandafter{\expandafter{\string#1}\expandafter
  \UD@secondoftwo\string}\expandafter\UD@firstoftwo\expandafter{\expandafter
  \UD@secondoftwo\string}\expandafter\expandafter\UD@firstoftwo{ }{}%
  \UD@secondoftwo}{\expandafter\expandafter\UD@firstoftwo{ }{}\UD@firstoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether argument's first token is a catcode-1-character
%%.............................................................................
%% \UD@CheckWhetherBrace{<Argument which is to be checked>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked has leading
%%                        catcode-1-token>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked has no leading
%%                        catcode-1-token>}%
\newcommand\UD@CheckWhetherBrace[1]{%
  \romannumeral0\expandafter\UD@secondoftwo\expandafter{\expandafter{%
  \string#1.}\expandafter\UD@firstoftwo\expandafter{\expandafter
  \UD@secondoftwo\string}\expandafter\expandafter\UD@firstoftwo{ }{}%
  \UD@firstoftwo}{\expandafter\expandafter\UD@firstoftwo{ }{}\UD@secondoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether brace-balanced argument starts with a space-token
%%.............................................................................
%% \UD@CheckWhetherLeadingSpace{<Argument which is to be checked>}%
%%                             {<Tokens to be delivered in case <argument
%%                               which is to be checked>'s 1st token is a
%%                               space-token>}%
%%                             {<Tokens to be delivered in case <argument
%%                               which is to be checked>'s 1st token is not
%%                               a space-token>}%
\newcommand\UD@CheckWhetherLeadingSpace[1]{%
  \romannumeral0\UD@CheckWhetherNull{#1}%
  {\expandafter\expandafter\UD@firstoftwo{ }{}\UD@secondoftwo}%
  {\expandafter\UD@secondoftwo\string{\UD@CheckWhetherLeadingSpaceB.#1 }{}}%
}%
\newcommand\UD@CheckWhetherLeadingSpaceB{}%
\long\def\UD@CheckWhetherLeadingSpaceB#1 {%
  \expandafter\UD@CheckWhetherNull\expandafter{\UD@secondoftwo#1{}}%
  {\UD@Exchange{\UD@firstoftwo}}{\UD@Exchange{\UD@secondoftwo}}%
  {\UD@Exchange{ }{\expandafter\expandafter\expandafter\expandafter
   \expandafter\expandafter\expandafter}\expandafter\expandafter
   \expandafter}\expandafter\UD@secondoftwo\expandafter{\string}%
}%
%%-----------------------------------------------------------------------------
%% Extract first inner undelimited argument:
%%
%%   \UD@ExtractFirstArg{ABCDE} yields  {A}
%%
%%   \UD@ExtractFirstArg{{AB}CDE} yields  {AB}
%%.............................................................................
\newcommand\UD@RemoveTillUD@SelDOm{}%
\long\def\UD@RemoveTillUD@SelDOm#1#2\UD@SelDOm{{#1}}%
\newcommand\UD@ExtractFirstArg[1]{%
  \romannumeral0%
  \UD@ExtractFirstArgLoop{#1\UD@SelDOm}%
}%
\newcommand\UD@ExtractFirstArgLoop[1]{%
  \expandafter\UD@CheckWhetherNull\expandafter{\UD@firstoftwo{}#1}%
  { #1}%
  {\expandafter\UD@ExtractFirstArgLoop\expandafter{\UD@RemoveTillUD@SelDOm#1}}%
}%
%%=============================================================================
%% \ReplaceEveryHash{<argument>}%
%%
%%   Each explicit catcode-6(parameter)-character-token of the <argument> 
%%   will be replaced by its stringification.
%%
%%   You obtain the result after two expansion-steps, i.e., 
%%   in expansion-contexts you get the result after "hitting" 
%%   \ReplaceEveryHash by two \expandafter.
%%   
%%   As a side-effect, the routine does replace matching pairs of explicit
%%   character tokens of catcode 1 and 2 by matching pairs of curly braces
%%   of catcode 1 and 2.
%%   I suppose this won't be a problem in most situations as usually the
%%   curly braces are the only characters of category code 1 / 2...
%%
%%   This routine needs \detokenize from the eTeX extensions.
%%-----------------------------------------------------------------------------
\newcommand\ReplaceEveryHash[1]{%
   \romannumeral0\UD@ReplaceEveryHashLoop{#1}{}%
}%
\newcommand\UD@ReplaceEveryHashLoop[2]{%
  \UD@CheckWhetherNull{#1}{ #2}{%
    \UD@CheckWhetherLeadingSpace{#1}{%
       \expandafter\UD@ReplaceEveryHashLoop
       \expandafter{\UD@removespace#1}{#2 }%
    }{%
      \UD@CheckWhetherBrace{#1}{%
        \expandafter\expandafter\expandafter\UD@PassFirstToSecond
        \expandafter\expandafter\expandafter{%
        \expandafter\UD@PassFirstToSecond\expandafter{%
            \romannumeral0\expandafter\UD@ReplaceEveryHashLoop
            \romannumeral0\UD@ExtractFirstArgLoop{#1\UD@SelDOm}{}%
        }{#2}}%
        {\expandafter\UD@ReplaceEveryHashLoop
         \expandafter{\UD@firstoftwo{}#1}}%
      }{%
       \expandafter\UD@CheckWhetherHash
       \romannumeral0\UD@ExtractFirstArgLoop{#1\UD@SelDOm}{#1}{#2}%
      }%
    }%
  }%
}%
\newcommand\UD@CheckWhetherHash[3]{%
    \expandafter\expandafter\expandafter\UD@CheckWhetherNull
    \expandafter\expandafter\expandafter{%
    \expandafter\UD@firstoftwo
    \expandafter{\expandafter}\string#1}{%
      \expandafter\expandafter\expandafter\UD@CheckWhetherNull
      \expandafter\expandafter\expandafter{%
      \expandafter\UD@firstoftwo
      \expandafter{\expandafter}\detokenize{#1}}{%
        \expandafter\UD@ReplaceEveryHashLoop
        \expandafter{\UD@firstoftwo{}#2}{#3#1}%
      }{%
        \expandafter\expandafter\expandafter\UD@PassFirstToSecond
        \expandafter\expandafter\expandafter{\expandafter\UD@Exchange
        \expandafter{\string#1}{#3}}{%
          \expandafter\UD@ReplaceEveryHashLoop
          \expandafter{\UD@firstoftwo{}#2}%
        }%
      }%
    }{%
      \expandafter\UD@ReplaceEveryHashLoop
      \expandafter{\UD@firstoftwo{}#2}{#3#1}%
    }%
}%
%----------------------------------------------------------------------

\newcommand\allowhash[1]{\ReplaceEveryHash{#1}}

\newcommand\allowanddetokenizehash[1]{%
  \detokenize\expandafter\expandafter\expandafter{\ReplaceEveryHash{#1}}%
}%

\makeatother

\begin{document}
\allowhash{One hash: #, not two}

\begingroup
\frenchspacing
\ttfamily \allowanddetokenizehash{One hash: #, not two. This time in braces:{#{#{#{#}}}#}}
\endgroup

For comparison the effect of \verb|\detokenize| without prior hash-replacing:

\begingroup
\frenchspacing
\ttfamily \detokenize{One hash: #, not two. This time in braces:{#{#{#{#}}}#}}
\endgroup

\end{document}

enter image description here

10

enter image description here

\documentclass{article}
\makeatletter
\def\allowhash{%%%%
    \bgroup\everyeof{\egroup}\catcode35=11\relax\scantokens}%
\makeatother
\begin{document}
\allowhash{One hash: #, not two}

\end{document}

or in a macro as requested in comments

enter image description here

\documentclass{article}
\makeatletter
\def\allowhash{%%%%
    \bgroup\catcode35=11\relax\afterassignment\egroup\gdef\foo}%
\makeatother
\begin{document}
\allowhash{One hash: #, not two}

[\foo]  [\foo]
\end{document}
  • What if I wanted the result in a macro instead of just printing it? – Andreas Storvik Strauman Sep 2 '18 at 11:35
  • @AndreasStorvikStrauman added a macro version – David Carlisle Sep 2 '18 at 11:39
  • Could you add some words of explanation how the first macro works? – siracusa Sep 2 '18 at 15:00
  • @siracusa it just makes # non-special without parsing the {...} as an argument of \allowhash, the \scantokens isn't really needed at all except the end of file hook is used to insert an endgroup to restore the meaning of # – David Carlisle Sep 2 '18 at 15:13
5

When \scantokens acts, the # characters are already doubled, because they're absorbed as the argument to a macro.

With the l3regex module of expl3 it's easier:

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn

\NewDocumentCommand{\allowhash}{m}
 {
  \tl_set:Nn \l_tmpa_tl { #1 }
  \regex_replace_all:nnN { \cP\# } { \cO\# } \l_tmpa_tl
  \tl_use:N \l_tmpa_tl
 }
\ExplSyntaxOff

\begin{document}

\allowhash{One hash: #, not two}

\end{document}

enter image description here

You can easily add support for saving the token list in a macro.

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn

\NewDocumentCommand{\allowhash}{om}
 {
  \tl_set:Nn \l_tmpa_tl { #2 }
  \regex_replace_all:nnN { \cP\# } { \cO\# } \l_tmpa_tl
  \IfNoValueTF { #1 }
   { \tl_use:N \l_tmpa_tl }
   { \tl_set_eq:NN #1 \l_tmpa_tl }
 }

\ExplSyntaxOff

\begin{document}

\allowhash{One hash: #, not two}

\allowhash[\foo]{One hash: #, not two}

\texttt{\meaning\foo}

\end{document}

enter image description here

(There seems to be a double space in the picture, but it's only due to \nonfrenchspacing.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.