# Define an active character inside a definition

\def\test{\catcode!=\active \def!{test}}


Of course the above code would not work because nothing is executed when TeX is reading the replacement text of a macro definition. ! gets catcode 12 anyway. I've tried a search on this site and someone suggested using \scantokens. As far as I know, \scantokens is an expandable command which reads the following token list, stores it in a pseudo-file and then inputs it, so the tokens get re-tokenized according to the new catcodes. However, the following code still fails and I can't figure out why.

\def\test{\catcode!=\active \scantokens{\def!}{test}}
\test


I get the error of "runaway definition".

The e-TeX reference defines \scantokens as follows:

\scantokens, when followed by a <general text>, decomposes the <balanced text> of the <general text> into the corresponding sequence of characters as if the <balanced text> were written unexpanded to a file; it then uses TeX's \input mechanism to re-process these characters under the current \catcode regime.

So the \def! part of your macro is processed as if it were defined in an extra file that is then \input into your main file. This includes the use of an internal end-fo-file marker when the whole <balanced text> has been processed.

In The TeXbook, p. 206, Knuth makes a note on file reading and \outer macros (thanks to GuM for pointing this out):

An \outer macro cannot appear in an argument (not even when \par is allowed), nor can it appear in the parameter text or the replacement text of a definition [...] The end of an input file or alignment template is also considered to be \outer in this sense; for example, a file shouldn't end in the middle of a definition.

This is exactly the problem you run into here. TeX is still scanning the definition of ! when the end of file after \scantokens occurs. It then complains about File ended while scanning definition of !. You can see the same behavior and error message if you replace \scantokens{\def!} by an actual \input were the input file includes only \def!.

Bruno Le Floch found a nice way to get around this problem by using a \let instead of \def to define !:

\def\bangdef{test}
\def\test{\catcode!=\active \scantokens{\let!}\bangdef}
\test


For completeness, here are two standard tricks that do not make use of \scantokens. One way to get the desired result is by defining \test in a regime which has the correct catcode already applied:

\begingroup
\catcode!=\active
\gdef\test{\catcode!=\active \def!{test}}
\endgroup
\test
\show!


Instead of the \gdef you could also save the current catcode of ! and restore it after the definition of \test.

Another common trick is to change the lowercase code of an active character, here ~, to the character code of the character to be made active, and then use \lowercase to apply this change:

\def\test{%
\begingroup
\lccode\~=\!
\lowercase{\endgroup \catcode\!=\active \def~}{test}%
}
\test
\show!

• On \scantokens, it is possible to avoid the issues with end-of-file but only by using an assignment. It's well-established that \scantokens is a bit 'fragile': it needs careful handling. – Joseph Wright Oct 6 '18 at 6:15
• @JosephWright Thanks, I updated the second example. Could you explain what kind of assignment you mean for the \scantokens approach? – siracusa Oct 6 '18 at 6:47
• Typically just \everyeof{\noexpand} but one can be more sophisticated (see the expl3 code for \tl_rescan:nn). – Joseph Wright Oct 6 '18 at 6:51
• As for a reference for the rule that forbids definitions to continue past the end of a file, see The TeXbook, p. 206, lines 7-13. If you want, I can insert the reference directly into your answer. – GuM Oct 6 '18 at 8:48
• Amusingly \scantokens{\let\foo}\bar works, so one way could be to \def\temporary{test} and \expandafter\let\scantokens{!}\temporary. – Bruno Le Floch Oct 6 '18 at 8:58

Use etex or pdftex.

\def\test{\catcode!=\active
\begingroup\endlinechar-1
\everyeof{{test}\endgroup}\scantokens{\gdef!}}

\test

\tt
\meaning!

\bye


Less original, and dropping the idea to end the scantokens argument before replacement text:

\def\test{\catcode!=\active
\scantokens{\def!{test}\relax}}% \relax to absorb an EOL space token

\tt

+++\test+++

\show!

\bye


Produces

> !=macro:
->test.
l.10 \show!

• The first method has the advantage that the definition of ! can easily be made to use itself with catcode 12, for example try with \everyeof{{!!!}\endgroup}. This will define the active ! to non-active !!!. – user4686 Oct 6 '18 at 21:45