Scantokens and Newlines

Question

I'm having problems with the usage of scantokens. I'm trying to convert a list that I read in while under the effects of \obeylines. For some reason, any time I use scantokens on my list, it stops reading at the ^^M.

Observe:

{\obeylines%
  \gdef\one{\message{foo}
    \message{bar}}
}
    \one
\expandafter\scantokens\expandafter{\one}
\bye

gives the output (./scan.tex foo bar foo )

As in, it won't put out the second bar.

This is a simplified example. But it could come up practically if you're reading in a paragraph line by line, and the ^^M ends up coming into your token stream because it was hidden by braces.

In any case, it's a bit inconvenient for \scantokens to just drop out on you like that.

Answer 1

The solution is to change the \newlinechar. This ensures that any ^^M that is in the stream to be read in gets interpreted as a newline rather than an end of line. An end of line character in the middle of a line is treating as if a comment character were there, and the rest of the line, until the newline character, is ignored.

If you add

\newlinechar=13 %that's the character code of newline

to the line before the scantokens, then you get the correct output.