So you asked several questions, but let me first answer your main question.
What happens here is that when the definition for
\foo is being parsed, it's identical to
\scantokens executes, it's as if it read the line
from a file with the current catcodes in effect. Since you reset
% to be a comment, the
\par B is ignored.
For your other questions, individual installations of TeX can determine what is treated as a new line when reading and writing files. If I recall correctly,
\n\r are treated as a newline for the purposes of input, at least that's what I recall from reading through the source for pdfTeX recently. For output, (i.e., writing files), I suspect it uses
\n on *NIX and
\r\n on Windows, but I haven't verified this.
After TeX reads a line of text from an input file—and before it begins to tokenize it—it removes all trailing space characters including
\n and appends the
\endlinechar character which is normally
\r). This happens regardless of there being a
% character in the line. When TeX encounters a
% character in its input (note that there is no comment token), it ignores the rest of the line, including the trailing
As far as I know,
\n) is not special in most contexts except it is often used as the
\newlinechar for use in
\scantokens. It is really treated like lines of input from a file, including the
\endlinechar at the end of each. For a simple example of this, try
(The space after the X is necessary because TeX looks for an optional space there.) Every line ends with a percent and yet the output shows
\scantokens has inserted that character.