# How are space tokens and empty lines processed by long commands (as their potential arguments)?

How are spaces and empty lines processed by long commands (i.e., those that do not accept paragraph breaks inside)? Are there different space tokens aside from " " and an empty line? It appears that an empty line counts as exactly one empty argument:

\documentclass{article}

\newcommand{\oneArg}[1]{}
\newcommand{\twoArgs}[2]{}
\newcommand{\threeArgs}[3]{}

\begin{document}

\indent
A \oneArg

B

% output:
% A B

A \twoArgs

B

C

% output:
% A
% C

A \threeArgs

B

C

% output:
% A C

\end{document}


And is there anything special one needs to know about math mode in this regard?

One pointer: Some relevant information is in this question, especially this discussion thread about \somecommand * being legal LaTeX.


\noPar{
A \threeArgs

B

C
}


the compiler will throw an error. As this is semantically not a paragraph break, the long-short distinction between commands should probably be described in terms of empty lines, not paragraph breaks. Or not?

-
That's exactly what I meant: an end of a line and a following empty line are equivalent to \par which is read in as an argument to your commands. Spaces alone are ignored when scanning for arguments except when they're used as delimiters for the arguments. (... or between delimiters of a delimited argument, or in braces { } ...) –  clemens Feb 19 '13 at 9:36
@cgnieder 1. Make it an answer? 2. Spaces are ignored or are not ignored "between delimiters of a delimited argument"? You mean "spaces are ignored between } and { of successive arguments' delimiters", correct? And the "in braces" addendum is a bit confusing (surely not all spaces occurring somewhere inside { } are ignored, perhaps only those at the beginning or end, but I though those are not ignored. –  Lover of Structure Feb 19 '13 at 9:40
I don't see the link to math mode: I removed the tag –  Joseph Wright Feb 19 '13 at 9:51
@LoverofStructure I disagree: TeX sees \noPar and so it absorbs its argument, which happens to contain \par. Error. –  egreg Feb 19 '13 at 11:39

The answer to the question in the title is technically "they are not processed" but I don't think that's the answer you want.

If you modify your definitions to

\newcommand{\oneArg}[1]{\long\def\a{[#1]}\typeout{\meaning\a}}
\newcommand{\twoArgs}[2]{\long\def\a{[#1][#2]}\typeout{\meaning\a}}
\newcommand{\threeArgs}[3]{\long\def\a{[#1][#2][#3]}\typeout{\meaning\a}}


you will see

\long macro:->[\par ]
\long macro:->[\par ][B]
\long macro:->[\par ][B][\par ]


Assuming normal catcodes are in force a blank line is turned by TeX into the token \par (literally the command name token \par not the primitive paragraph end function) It does this at a very early stage as characters are being tokenised, so before any token lists are passed to a macro. So a macro never sees a blank line in its argument. The behaviour is always as if you replace the blank line by \par in the input file.

Space tokens are similarly processed at this early stage. Spaces at the end of the line and the beginning of the next are discarded and never tokenised at all so macros have no record of them. (You can not prevent the discarding of space at the end of the line even if you change the catcode of space) and runs of spaces characters only produce one space token. It is the tokens not the file characters that are passed to a macro.

If you have non-delimited arguments as in your example any spaces tokens are skipped while looking for the argument, if you want a space to be the argument you need { }. \par can be an undelimited argument if the macro is \long or an error otherwise.

-

TeX's tokenization process is important here. When (La)TeX reads one newline

some text
some more text


it converts the newline to a space (with any spaces at the start of the second line ignored): some text some more text. However, when TeX reads two consecutive newlines it converts to a \par token

some text

some more text


ends up as some text\par some more text. (I'm assuming standard setting for \endlinechar here.)

Grabbing arguments happens after this process has occurred, so you are not seeing 'blank lines' being read as arguments but rather \par tokens. It's \par tokens in the argument of a command that make the difference between 'short' and 'long' commands, and these have to appear directly in the argument to raise an error with a 'short' command.

-
So there's a difference between "token \par" and "macro name token \par" ? –  yo' Feb 19 '13 at 9:53
@tohecz None: what there is a difference between is a token called \par (a macro, \let to a something, ...) and the \par primitive used by TeX internally. See for example \endgraf, which is \let to the \par primitive but is allowed in 'short' arguments as it is not a \par token. –  Joseph Wright Feb 19 '13 at 10:02
So I still don't understand, at which stage TeX recognizes the difference between \section{x\par y} and \section{x<EOL><EOL>y} so that the 1st goes through, but the 2nd throws an error? –  yo' Feb 19 '13 at 10:07
@tohecz You never get \section{x<EOL><EOL>y}: TeX tokenizes as it reads the argument, so you can only get \section{x\par y} (unless you do verbatim-like reading by fiddling with \endlinechar). –  Joseph Wright Feb 19 '13 at 10:11
@LoverofStructure TeX is in 'skipping spaces' mode after the first newline, so any intervening spaces are ignored. The TeXbook is the place to read up on this. –  Joseph Wright Feb 19 '13 at 10:27

(1) White space not containing two <end-of-line>s is skipped when scanning for arguments. (2) White-space containing at least two <end-of-lines> is converted to \par and can be passed as an argument.


As well \par in case (2) is a valid not only as an argument, but as an argument delimiter, too. However, the argument will still contain a terminating space token:

\def\xy#1\par{x#1y}

Hello \xy World

How are you

\bye


Output (one line, remember we "ate" the \par using it as a argument delimiter):

Hello xWorld yHow are you

AFAIK this is used in ConTeXt sometimes and it allows you to get rid of many braces, e.g. consider the defintion \def\Section#1\par{\section{#1}}, the your code would be:

\Section Your well-being

How are you?


Output: