5

I working in a context where I'm changing catcodes and need to rescan the tokens before proceeding to process the text. But, I'm running into problems. Even with this bare minimum code, I still have issues.

\documentclass{article}

\makeatletter
\newcommand\aeinput[1]{%%
  \begingroup
    \scantokens{\gdef\ae@input{::#1::}}%%
  \endgroup
  \ae@input
  }%%
\makeatother

\begin{document}
\aeinput{HELLO}
\end{document}

Resulting in the error:

! Undefined control sequence.
\aeinput ...ae@input {::#1::}}\endgroup \ae@input 

I get the same error if I write:

\scantokens{\xdef\ae@input{::#1::}}%%

Why is this not compiling?

Also, if I do something like

\expandafter\gdef\expandafter\ae@input\expandafter{\scantokens{::#1::}}%%

I get the following error:

! File ended while scanning definition of \ae@input.

Even adding \everyoef{\noexpand} doesn't help.

HOWEVER, the following does work (which makes me even more confused).

\xdef\ae@input{\scantokens{::#1::}}%%

So, what's happening here?

Where does this question come from?

I thought I had a simple answer for inputing text within an environment for which I've already changed the catcode of . Basically, in that question I have an environment

\begin{speech}
   inside here periods have a different catcode

 \end{speech}

The OP for that question wanted to do something like

\begin{speech}
  \input{text.tex}
\end{speech}

Clearly, \input is not going to work here because . has the wrong catcode. I thought I could easily remedy this (and allow the OP to continue writing \input{text.tex}) by doing something like

\makeatletter
\newcommand\aeinput[1]{%%
  \begingroup
  \catcode`\.=\periodcatcode
  \scantokens{\gdef\ae@input{\input{#1}}}%%
  \endgroup
  \ae@input
  }%%
\makeatother

An alternative approach would have been to do something like

\makeatletter
\newcommand\aeinput[1]{%%
  \begingroup
  \catcode`\.=\periodcatcode
  \scantokens{\gdef\ae@filehandle{#1}}%%
  \endgroup
  \input{\ae@filehandle}%%
  }%%
\makeatother

That way, the input file would handle periods appropriately according to the speech environment.

Something like

\newcommand\aeinput[1]{%%
  \begingroup
  \catcode`\.=\periodcatcode
  \scantokens{\input{#1}}%%
  \endgroup
}

Only superficially looks like it could work, but the input file would have not have the correct catcode for the periods anymore.

8
  • What is the purpose of \xdef? This will also fully expand #1. Why not just \newcommand\aeinput[1]{\scantokens{#1\ignorespaces}}? Oct 3 '18 at 5:06
  • @HenriMenke According to the etex documentation \scantokens is expandable. But, let's say it's not. Why in the world would \xdef\ae@input{\scantokens{....}} be the only instance for which my code compiled?
    – A.Ellett
    Oct 3 '18 at 5:10
  • @HenriMenke You changed your comment before I noticed.
    – A.Ellett
    Oct 3 '18 at 5:11
  • Have you had a look at tex.stackexchange.com/questions/117906/… ? Oct 3 '18 at 5:15
  • @HenriMenke Yes, I have and I can't seem to make things work. I've updated my answer to explain why your first suggestion is not tenable.
    – A.Ellett
    Oct 3 '18 at 5:20
6

Remember that \scantokens applies the category codes in force when it's used. Thus in

\documentclass{article}

\makeatletter
\newcommand\aeinput[1]{%%
  \begingroup
    \scantokens{\gdef\ae@input{::#1::}}%%
  \endgroup
  \ae@input
  }%%
\makeatother

\begin{document}
\aeinput{HELLO}
\end{document}

the \scantokens line is inserted in the doucment, where @ is not a letter. So \gdef\ae@input defines \ae as a macro to be followed by @input with replacement text ::<#1>::. You need to change the category codes inside your group

\documentclass{article}

\makeatletter
\newcommand\aeinput[1]{%%
  \begingroup
    \makeatletter
    \scantokens{\gdef\ae@input{::#1::}}%%
  \endgroup
  \ae@input
  }%%
\makeatother

\begin{document}
\aeinput{HELLO}
\end{document}

As you observe, whilst \scantokens is expandable, getting that right is a bit tricky as it has some oddities about the 'end of file' for the virtual file it uses. Those can be got around (see the \tl_rescan:nn code and similar in expl3), but it's non-trivial.

1
  • OMG. How did I not see that! Yikes. Subtle, but also kind of obvious---at least for the amount of monkeying around I do with this stuff. Thank you.
    – A.Ellett
    Oct 3 '18 at 6:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.