I'm having problems with the usage of scantokens. I'm trying to convert a list that I read in while under the effects of \obeylines. For some reason, any time I use scantokens on my list, it stops reading at the ^^M.



gives the output (./scan.tex foo bar foo )

As in, it won't put out the second bar.

This is a simplified example. But it could come up practically if you're reading in a paragraph line by line, and the ^^M ends up coming into your token stream because it was hidden by braces.

In any case, it's a bit inconvenient for \scantokens to just drop out on you like that.


1 Answer 1


The solution is to change the \newlinechar. This ensures that any ^^M that is in the stream to be read in gets interpreted as a newline rather than an end of line. An end of line character in the middle of a line is treating as if a comment character were there, and the rest of the line, until the newline character, is ignored.

If you add

\newlinechar=13 %that's the character code of newline

to the line before the scantokens, then you get the correct output.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.