So you asked several questions, but let me first answer your main question.
What happens here is that when the definition for \foo
is being parsed, it's identical to
\def\foo{\scantokens{A%\par B}}
Now when \scantokens
executes, it's as if it read the line
A%\par B
from a file with the current catcodes in effect. Since you reset %
to be a comment, the \par B
is ignored.
For your other questions, individual installations of TeX can determine what is treated as a new line when reading and writing files. If I recall correctly, \n
, \r
, \r\n
, and \n\r
are treated as a newline for the purposes of input, at least that's what I recall from reading through the source for pdfTeX recently. For output, (i.e., writing files), I suspect it uses \n
on *NIX and \r\n
on Windows, but I haven't verified this.
After TeX reads a line of text from an input file—and before it begins to tokenize it—it removes all trailing space characters including \r
and \n
and appends the \endlinechar
character which is normally ^^M
(i.e., \r
). This happens regardless of there being a %
character in the line. When TeX encounters a %
character in its input (note that there is no comment token), it ignores the rest of the line, including the trailing \r
.
As far as I know, ^^J
(i.e., \n
) is not special in most contexts except it is often used as the \newlinechar
for use in \write
.
I forgot \scantokens
. It is really treated like lines of input from a file, including the \endlinechar
at the end of each. For a simple example of this, try
\endlinechar`X %
\scantokens{A}%
\bye%
(The space after the X is necessary because TeX looks for an optional space there.) Every line ends with a percent and yet the output shows AX
because \scantokens
has inserted that character.
\scantokens
. Changing the catcode deals with the tokenization of the replacement text of\foo
. – TH. Jan 17 '11 at 13:28