How are space tokens and empty lines processed by long commands (as their potential arguments)?

Question

How are spaces and empty lines processed by long commands (i.e., those that do not accept paragraph breaks inside)? Are there different space tokens aside from " " and an empty line? It appears that an empty line counts as exactly one empty argument:

\documentclass{article}

\newcommand{\oneArg}[1]{}
\newcommand{\twoArgs}[2]{}
\newcommand{\threeArgs}[3]{}

\begin{document}

\indent
A \oneArg

B

% output:
% A B


A \twoArgs

B

C

% output:
% A
% C

A \threeArgs

B

C

% output:
% A C

\end{document}

And is there anything special one needs to know about math mode in this regard?

One pointer: Some relevant information is in this question, especially this discussion thread about \somecommand * being legal LaTeX.

---

Addendum: An interesting detail about short macros (those defined for example with \newcommand*): If I add \newcommand*{\noPar}[1]{#1} to my source code and try to compile an additional codechunk

\noPar{
A \threeArgs

B

C
}

the compiler will throw an error. As this is semantically not a paragraph break, the long-short distinction between commands should probably be described in terms of empty lines, not paragraph breaks. Or not?

Answer 1

The answer to the question in the title is technically "they are not processed" but I don't think that's the answer you want.

If you modify your definitions to

\newcommand{\oneArg}[1]{\long\def\a{[#1]}\typeout{\meaning\a}}
\newcommand{\twoArgs}[2]{\long\def\a{[#1][#2]}\typeout{\meaning\a}}
\newcommand{\threeArgs}[3]{\long\def\a{[#1][#2][#3]}\typeout{\meaning\a}}

you will see

\long macro:->[\par ]
\long macro:->[\par ][B]
\long macro:->[\par ][B][\par ]

Assuming normal catcodes are in force a blank line is turned by TeX into the token \par (literally the command name token \par not the primitive paragraph end function) It does this at a very early stage as characters are being tokenised, so before any token lists are passed to a macro. So a macro never sees a blank line in its argument. The behaviour is always as if you replace the blank line by \par in the input file.

Space tokens are similarly processed at this early stage. Spaces at the end of the line and the beginning of the next are discarded and never tokenised at all so macros have no record of them. (You can not prevent the discarding of space at the end of the line even if you change the catcode of space) and runs of spaces characters only produce one space token. It is the tokens not the file characters that are passed to a macro.

If you have non-delimited arguments as in your example any spaces tokens are skipped while looking for the argument, if you want a space to be the argument you need { }. \par can be an undelimited argument if the macro is \long or an error otherwise.

Answer 2

TeX's tokenization process is important here. When (La)TeX reads one newline

some text
some more text

it converts the newline to a space (with any spaces at the start of the second line ignored): some text some more text. However, when TeX reads two consecutive newlines it converts to a \par token

some text

some more text

ends up as some text \par some more text. (I'm assuming standard setting for \endlinechar here.)

Grabbing arguments happens after this process has occurred, so you are not seeing 'blank lines' being read as arguments but rather \par tokens. It's \par tokens in the argument of a command that make the difference between 'short' and 'long' commands, and these have to appear directly in the argument to raise an error with a 'short' command.

Answer 3

(1) White space not containing two <end-of-line>s is skipped when scanning for arguments. (2) White-space containing at least two <end-of-lines> is converted to \par and can be passed as an argument.

On the other hand, when TeX scans a definition of a command without \long, and spots \par (either as \par or as (2) above), it throws an error. This is so that you can easier debug wrong grouping. (Remember: \newcommand => yes-\long and \newcommand* => no-\long.)

---

As well \par in case (2) is a valid not only as an argument, but as an argument delimiter, too. However, the argument will still contain a terminating space token:

\def\xy#1\par{x#1y}

Hello \xy World

How are you

\bye

Output (one line, remember we "ate" the \par using it as a argument delimiter):

Hello xWorld yHow are you

---

AFAIK this is used in ConTeXt sometimes and it allows you to get rid of many braces, e.g. consider the defintion \def\Section#1\par{\section{#1}}, the your code would be:

\Section Your well-being

How are you?

Output:

Your well-being

How are you?