# Use of \everyeof and \endlinechar with \scantokens

The e-TeX \scantokens primitive allows retokenization of input. However, it is almost always used in a group where \everyeof and \endlinechar are set (see for example Can I convert a string to catcode 11?). What is the reasoning for requiring these two steps?

The \scantokens primitive is described in the e-TeX manual as working in a similar manner to the following code:

\toks0={...}% '...' is the rescanned material
\immediate\openout0=file
\immediate\write0{\the\toks0}
\immediate\closeout0
\input file


but without the use of files and in an expandable manner. However, it does use the some of the same internals as the above. This has a consequence for using the primitive.

A pseudo-file is 'read' by TeX, and this is treated as having an end-of-file (EOF) marker. \scantokens tries to read this as a token, but that will raise an error, for example

! File ended while scanning definition of \demo


with code

\edef\demo{\scantokens{foo}}


To prevent this, you need to set \everyeof to insert a \noexpand before this marker:

\everyeof{\noexpand}
\edef\demo{\scantokens{foo}}


TeX then does not try to read past the end of the file and this error is avoided.

The second issue is that TeX tokenizes the 'end of line' characters in the normal way inside \scantokens. The common use is to have a single line scanned, as above, but the result will not be as might be expected:

\everyeof{\noexpand}
\edef\demo{\scantokens{foo}}
\show\demo


yields

> \demo=macro:
->foo .


with an additional space: the final 'end of line' (end of the pseudo-file) is converted to a space. To prevent this, you normally alter the end-of-line behaviour with

\endlinechar=-1


so that the end-of-line is ignored and no space is added.

It's then standard to wrap everything up in a group, for example when saving the result in a macro

\long\def\safescantokens#1#2{%
\begingroup
\everyeof{\noexpand}%
\endlinechar=-1
\xdef#1{\scantokens{#2}}%
\endgroup
}
\safescantokens\demo{foo}


The group is used here so that the two additional steps don't affect any other code, while \xdef is the simplest way to get the result outside of the group. (An appropriate \expandafter chain is also a possible approach for that.)

All of this makes the resulting use non-expandable, which somewhat defeats the point of the primitive (although files are still not used). As a result, in LuaTeX there is a \scantextokens primitive which specifically addresses these issues: the end-of-file is ignored and no end line character is inserted after the last line (which is almost always the only line).

• Is this the core of what \tl_set_rescan:Nnn does? Jun 6 '13 at 11:19
• @cgnieder Partly, but there is more 'cleverness' inside the code to cover the case where you are not carrying out an \edef or similar. Jun 6 '13 at 12:49
• That is so weird. What possible benefit could there be to directly emulating the behavior of the non-eTeX hack when writing a new primitive that presumably has access to all the internals of tex? Even if the file-reading method is necessary to make tex do the rescan, why leave its detritus in the input stream? Jun 6 '13 at 13:51
• @RyanReich If you look at some of the other e-TeX expandable primitives (\detokenize and \unexpanded come to mind) the way they are described is very much 'as you could do using a toks but without the assignment'. You'd have to ask the NTS people for a definitive answer, but my guess is that the idea was to make as few changes as possible while adding new abilities. As I hope I indicate, that doesn't come off so well in \scantokens (although see David's answer). I find the entire primitive rather risky, to be honest, so don't tend to use it. Jun 6 '13 at 13:56
• @JosephWright I've been mostly using \scantokens{...\noexpand}. I wonder whether that catches both, the EOF marker and the EOL character. Because after reading \noexpand TeX should be skipping spaces, so EOL should be discarded and \noexpand should be applied to EOF. Does that sound correct? Aug 28 '19 at 23:34

An alternative, and not totally unsafe, definition is

\long\def\safescantokensb#1#2{%
\scantokens{\def#1{#2}\ignorespaces}%
}
\safescantokensb\demo{foo}

• Not a bad idea: I wonder if it's general enough to replace the approach I'd highlighted Jun 6 '13 at 13:57