# \scantokens not playing nice with \def or \gdef etc

I working in a context where I'm changing catcodes and need to rescan the tokens before proceeding to process the text. But, I'm running into problems. Even with this bare minimum code, I still have issues.

\documentclass{article}

\makeatletter
\newcommand\aeinput[1]{%%
\begingroup
\scantokens{\gdef\ae@input{::#1::}}%%
\endgroup
\ae@input
}%%
\makeatother

\begin{document}
\aeinput{HELLO}
\end{document}


Resulting in the error:

! Undefined control sequence.
\aeinput ...ae@input {::#1::}}\endgroup \ae@input


I get the same error if I write:

\scantokens{\xdef\ae@input{::#1::}}%%


Why is this not compiling?

Also, if I do something like

\expandafter\gdef\expandafter\ae@input\expandafter{\scantokens{::#1::}}%%


I get the following error:

! File ended while scanning definition of \ae@input.


Even adding \everyoef{\noexpand} doesn't help.

HOWEVER, the following does work (which makes me even more confused).

\xdef\ae@input{\scantokens{::#1::}}%%


So, what's happening here?

Where does this question come from?

I thought I had a simple answer for inputing text within an environment for which I've already changed the catcode of . Basically, in that question I have an environment

\begin{speech}
inside here periods have a different catcode

\end{speech}


The OP for that question wanted to do something like

\begin{speech}
\input{text.tex}
\end{speech}


Clearly, \input is not going to work here because . has the wrong catcode. I thought I could easily remedy this (and allow the OP to continue writing \input{text.tex}) by doing something like

\makeatletter
\newcommand\aeinput[1]{%%
\begingroup
\catcode\.=\periodcatcode
\scantokens{\gdef\ae@input{\input{#1}}}%%
\endgroup
\ae@input
}%%
\makeatother


An alternative approach would have been to do something like

\makeatletter
\newcommand\aeinput[1]{%%
\begingroup
\catcode\.=\periodcatcode
\scantokens{\gdef\ae@filehandle{#1}}%%
\endgroup
\input{\ae@filehandle}%%
}%%
\makeatother


That way, the input file would handle periods appropriately according to the speech environment.

Something like

\newcommand\aeinput[1]{%%
\begingroup
\catcode\.=\periodcatcode
\scantokens{\input{#1}}%%
\endgroup
}


Only superficially looks like it could work, but the input file would have not have the correct catcode for the periods anymore.

• What is the purpose of \xdef? This will also fully expand #1. Why not just \newcommand\aeinput[1]{\scantokens{#1\ignorespaces}}? Oct 3 '18 at 5:06
• @HenriMenke According to the etex documentation \scantokens is expandable. But, let's say it's not. Why in the world would \xdef\ae@input{\scantokens{....}} be the only instance for which my code compiled? Oct 3 '18 at 5:10
• @HenriMenke You changed your comment before I noticed. Oct 3 '18 at 5:11
• Have you had a look at tex.stackexchange.com/questions/117906/… ? Oct 3 '18 at 5:15
• @HenriMenke Yes, I have and I can't seem to make things work. I've updated my answer to explain why your first suggestion is not tenable. Oct 3 '18 at 5:20

Remember that \scantokens applies the category codes in force when it's used. Thus in

\documentclass{article}

\makeatletter
\newcommand\aeinput[1]{%%
\begingroup
\scantokens{\gdef\ae@input{::#1::}}%%
\endgroup
\ae@input
}%%
\makeatother

\begin{document}
\aeinput{HELLO}
\end{document}


the \scantokens line is inserted in the doucment, where @ is not a letter. So \gdef\ae@input defines \ae as a macro to be followed by @input with replacement text ::<#1>::. You need to change the category codes inside your group

\documentclass{article}

\makeatletter
\newcommand\aeinput[1]{%%
\begingroup
\makeatletter
\scantokens{\gdef\ae@input{::#1::}}%%
\endgroup
\ae@input
}%%
\makeatother

\begin{document}
\aeinput{HELLO}
\end{document}


As you observe, whilst \scantokens is expandable, getting that right is a bit tricky as it has some oddities about the 'end of file' for the virtual file it uses. Those can be got around (see the \tl_rescan:nn code and similar in expl3`), but it's non-trivial.

• OMG. How did I not see that! Yikes. Subtle, but also kind of obvious---at least for the amount of monkeying around I do with this stuff. Thank you. Oct 3 '18 at 6:11