Introduction to the topic
Chapter 24: Summary of Vertical Mode of the TeXbook, the chapter where TeX's parts of speech are defined in a modified form of the
Backus-Naur-notation, indicates that the pattern for defining macros by means of primitives \def, \gdef, \edef, \xdef is:
⟨prefixes⟩\def|\gdef|\edef|\xdef⟨control sequence⟩⟨parameter text⟩⟨left brace⟩⟨balanced text⟩⟨right brace⟩
⟨prefixes⟩ either is empty or is a combination of some or of all of the primitives\global/\outer/\long/\protected.
⟨control sequence⟩ is a control word token or a control symbol token or an active character token.
⟨left brace⟩ is an explicit character token of category 1 (begin group).
⟨right brace⟩ is an explicit character token of category 2 (end group).
In Chapter 20: Definitions (also called Macros) of the TeXbook the ⟨balanced text⟩ is also referred to as ⟨replacement text⟩.
⟨parameter text⟩ contains no ⟨left brace⟩ and no ⟨right brace⟩ and there can be as many as nine parameters, #1 to #9, which must be numbered in order. # stands for any explicit character token of category 6 (parameter). At the time of defining macros, parameters in the ⟨parameter text⟩ can be delimited in quite a general way by placing tokens behind them.
Only parameters introduced in the ⟨parameter text⟩ may occur in the ⟨replacement text⟩.
Providing parameters at the time of defining a macro implies that at the time of expanding an instance of the macro in question and delivering the replacement text TeX for each parameter gathers a set of tokens from the token stream and within the ⟨replacement text⟩ replaces the corresponding parameter by the corresponding set of tokens.
A set of tokens gathered for replacing a parameter of a ⟨replacement text⟩ is called an argument/a macro argument.
Thus the ⟨parameter text⟩ of a macro definition can be seen as a pattern which is to be obeyed by TeX at the time of expanding instances of the macro in question when gathering from the token stream sets of tokens forming that instance's arguments.
At the time of defining a macro you can have ⟨parameter text⟩ and parameters.
At the time of expanding an instance of a macro it is about macro arguments.
If in the ⟨parameter text⟩ of a macro's ⟨definition⟩ a parameter is delimited, then at the time of expanding an instance of that macro, the corresponding macro argument, whose tokens are to be gathered from the token stream, is delimited also, by the set of tokens which in the ⟨parameter text⟩ forms the delimiter of the corresponding parameter.
Subtlety:
In this answer "⟨replacement text⟩" (with "⟨" and "⟩") denotes the ⟨balanced text⟩ at the time of defining a macro. ⟨replacement text⟩ may contain parameters #1, #2, ..., #9 that are introduced in the ⟨parameter text⟩.
In this answer "replacement text" (without "⟨" and "⟩") denotes the tokens which you get at the time of expanding an instance of a macro. In "replacement text" parameters are replaced by the tokens that form the corresponding macro arguments.
The focus of this answer is not on providing ⟨parameter text⟩ when defining macros.
The focus of this answer is on how at the time of expanding instances of macros TeX gathers sets tokens from the token stream that are to form macro arguments.
Can \outer tokens occur in ⟨parameter text⟩?
Can \outer tokens be components of macro arguments?
Tokens that are \outer cannot be part of a ⟨parameter text⟩.
\outer tokens also cannot be components of macro arguments.
\outer tokens can make it into the ⟨replacement text⟩ of macro definitions by means of the \edef...\noexpand-trick :
\outer\def\foo{This is foo and foo is outer.}
\edef\bar{%
This is bar and it calls
{\noexpand\tt\noexpand\string\noexpand\foo:}
\noexpand\foo
}
\foo
\bar
%This would yield an error:
% ! Forbidden control sequence found while scanning definition of ...
%\def\bar{This is bar and it calls {\tt\string\foo:} \foo}
\bye
About \long and short
In case a macro is defined in terms of \long, such as \long\def\macro#1#2{...}, all arguments of instances of that macro can contain the token \par.
A macro defined in terms of \long colloquially is called a "long macro".
In case a macro is not defined in terms of \long, such as \def\macro#1#2{...}, if with one of its instances an argument contains the token \par, this triggers an error message on the console/terminal in terms of a question about "Runaway argument?"
E.g., something like the following console-output:
*\def\macro#1#2{Argument 1: #1; Argument 2: #2}
*\macro{\par argument 1}{argument 2}
Runaway argument?
{
! Paragraph ended before \macro was complete.
<to be read again>
\par
<*> \macro{\par
argument 1}{argument 2}
?
Unlike the \long-prefix, there is no \short-prefix. Nonetheless, a macro not defined in terms of \long is sometimes colloquially called a "short macro".
The \long mechanism detects the token \par no matter if \par has its usual meaning (as something that "tells" TeX that the end of the paragraph is reached and that TeX shall typeset that paragraph) or if \par is redefined. The \long-mechanism does not detect tokens other than \par whose meaning equals that of the \par-primitive.
E.g., after
\let\pur=\par
\def\par{Hello!}
the \long-mechanism still takes \par as a trigger for the runaway-argument-message while \pur is not a trigger for the runaway-argument-message.
About the order in time in which TeX does gather macro arguments
When expanding an instance of a macro, TeX takes the ⟨parameter text⟩ from the corresponding ⟨definition⟩ for the directive to gather the macro arguments one by one from the token stream.
For example, with a definition
\def\macro#1#2\DelimiterOne#3#4\DelimiterTwo\DelimiterTwo{%
⟨replacement text where parameters #1/#2/#3/#4 may occur⟩%
}
the ⟨parameter text⟩ is: #1#2\DelimiterOne#3#4\DelimiterTwo\DelimiterTwo.
The parameter #1 is not delimited.
The parameter #2 is delimited by the token \DelmiterOne.
The parameter #3 is not delimited.
The parameter #4 is delimited by the tokens \DelmiterTwo\DelimiterTwo.
Thus at the time of expanding an instance of \macro TeX first gathers from the token-stream a set of tokens according to the rules for gathering an undelimited argument. That set of tokens is the first argument of this instance of \macro. When delivering the replacement text, TeX replaces the parameter #1 of \macro's ⟨replacement text⟩ by the set of tokens forming the first argument of this instance of \macro.
Then TeX gathers from the token-stream a set of tokens according to the rules for gathering a delimited argument whereby the delimiter is formed by the token \DelimiterOne. That set of tokens is the second argument of this instance of \macro. When delivering the replacement text, TeX replaces the parameter #2 of \macro's ⟨replacement text⟩ by the set of tokens forming the second argument of this instance of \macro.
Then TeX gathers from the token-stream a set of tokens according to the rules for gathering an undelimited argument. That set of tokens is the third argument of this instance of \macro. When delivering the replacement text, TeX replaces the parameter #3 of \macro's ⟨replacement text⟩ by the set of tokens forming the third argument of this instance of \macro.
Then TeX gathers from the token-stream a set of tokens according to the rules for gathering a delimited argument whereby the delimiter is formed by the tokens \DelimiterTwo\DelimiterTwo. That set of tokens is the fourth argument of this instance of \macro. When delivering the replacement text, TeX replaces the parameter #4 of \macro's ⟨replacement text⟩ by the set of tokens forming the fourth argument of this instance of \macro.
There are two sorts of macro arguments:
1. Undelimited macro arguments
Undelimited macro arguments are indicated by undelimited parameters in the ⟨parameter text⟩. An undelimited parameter is a parameter whereafter either the ⟨left bracet⟩ enclosing the ⟨replacement text⟩ or another parameter follows.
There are two possibilities what a set of tokens forming an undelimited argument can look like:
A single token which neither is an explicit character token of category 1(begin group) nor is an explicit character token of category 2(end group). That is, a single token which neither is an opening curly brace {1, nor is a closing curly brace }2.
Henceforth such a thing is called a not-enclosed undelimited argument.
A set of tokens enclosed in a leading explicit character token of category 1(begin group) and a trailing explicit character token of category 2(end group), and containing explicit character tokens of category 1(begin group) and explicit character tokens of category 2(end group) in a way where each explicit character token of category 1(begin group) has exactly one matching explicit character token of category 2(end group) and vice versa. That is, it is a set of tokens enclosed in an opening curly brace and a closing curly brace, and if there are more curly braces within, then each opening curly brace has exactly one matching closing curly brace and each closing curly brace has exactly one matching opening curly brace.
Henceforth such a thing is called an enclosed undelimited argument.
Using this terminology you can say: Enclosed undelimited arguments consist of stuff that forms the enclosing and of stuff which is enclosed. (The set of tokens forming stuff which is enclosed may be empty.)
At the time of expanding a macro the tokens forming the replacement text of that macro are delivered. Hereby undelimited parameters (#1 or #2 or...) occurring in the ⟨replacement text⟩ are replaced as follows:
In case the undelimited parameter happens to be a not-enclosed undelimited argument, the parameter is replaced by the token forming that not-enclosed undelimited argument.
In case the undelimited parameter happens to be an enclosed undelimited argument, the parameter is replaced by the tokens that form the enclosed stuff of the enclosed undelimited argument, while the explicit character token of category 1(begin group) and the explicit character token of category 2(end group) forming the enclosing are discarded/removed/stripped off.
If a macro passes an argument to another macro, something like
\def\foo#1{\bar#1!}
\def\bar#1{\string\bar's argument is in parentheses: (#1)}
, where the exclamation mark indicates the place of the last token of the argument of \foo and parentheses surround what is processed as argument of \bar, then each macro might remove a level of surrounding curly braces:
\foo{{Argument}} yields \bar{Argument}!, which in turn yields \string\bar's argument is in parentheses: (Argument)!.
\foo{{A}rgument} yields \bar{A}rgument!, which in turn yields \string\bar's argument is in parentheses: (A)rgument!.
\foo{Argument} yields \bar Argument!, which in turn yields \string\bar's argument is in parentheses: (A)rgument!.
When TeX starts gathering an undelimited macro argument from the token stream, it silently discards explicit space tokens (i.e., explicit character tokens of category 10(space) and character code 32), until finding a token that is not an explicit space token.
That token may be something other than an explicit character token of category 1(begin group) or 2(end group) and thus form a not-enclosed undelimited macro argument.
That token may as well be an explicit character token of category 1(begin group) and thus form the left side of the enclosing of an enclosed undelimited macro argument.
This discarding of explicit space tokens is the reason why having spaces between/right before undelimited macro arguments and not having spaces between/right before undelimited macro arguments makes no difference:
\def\ProcessTwoNonDelimitedArgs#1#2{Arg1:(#1), Arg2:(#2)}
% A <space token> between {Argument1} and {Argument2}:
\ProcessTwoNonDelimitedArgs{Argument1} {Argument2}
% No <space token> between {Argument1} and {Argument2}:
\ProcessTwoNonDelimitedArgs{Argument1}{Argument2}
\bye
2. Delimited macro arguments
Delimited macro arguments are indicated by delimited parameters in the ⟨parameter text⟩. A delimited parameter is a parameter whereafter in the ⟨parameter text⟩ a set of tokens follows neither containing explicit character tokens of category 1(begin group), nor containing explicit character tokens of category 2(end group), nor containing explicit character tokens of category 6(parameter). Such a set of tokens is called the delimiter of the parameter.
A delimited macro argument is a macro argument which is delimited by an instance of the set of tokens which in the ⟨parameter text⟩ of the ⟨definition⟩ of the macro forms the delimiter of the corresponding parameter.
I.e., a delimited macro argument is delimited by a set of tokens containing neither opening curly braces, closing curly braces, nor explicit hashes.
(#6, hash, is an explicit character token of category 6(parameter). Examples of implicit character tokens of category 6(parameter) are the implicit hash-character-tokens of category 6(parameter) \hash and U with \let\hash=# or \catcode`\U=13 \letU=#. The #-doubling at times of unexpanded writing etc only occurs with explicit character tokens of category 6(parameter).)
With delimited parameters the delimiting set of tokens is always right behind the parameter. With delimited macro arguments the delimiting set of tokens is always right behind the macro argument.
Argument delimiters can contain implicit character tokens of category 6(parameter) if the tokens constituting the implicit character tokens of category 6(parameter) were not yet implicit character tokens of category 6(parameter) when defining took place, but were turned into implicit character tokens of category 6(parameter) by redefining them afterwards.
\let\ImplicitHash=# %
\def\Macro#1\ImplicitHash\ImplicitHash#2\ImplicitHash{Arg1:(#1), Arg2:(#2)}
\Macro{A1}\ImplicitHash\ImplicitHash{A2}\ImplicitHash
\bye
does not work, while
\def\Macro#1\ImplicitHash\ImplicitHash#2\ImplicitHash{Arg1:(#1), Arg2:(#2)}
\let\ImplicitHash=# %
\Macro{A1}\ImplicitHash\ImplicitHash{A2}\ImplicitHash
\bye
does work.
When TeX gathers a delimited macro argument for a macro, tokens are gathered from the token stream until
- either tokens are found that form the matching argument delimiter,
- or it is obvious that an error message needs to be raised.
At the time of gathering tokens of a delimited macro argument, a sequence of tokens which looks like those tokens that within the ⟨parameter text⟩ of the corresponding macro definition form the argument delimiter, is only considered the argument-delimiter if it occurs within the same brace-group as the macro token where the delimited argument in question belongs to.
I.e.: suppose there is a sequence of tokens in the token stream which at first looks like the tokens that form the argument delimiter within the ⟨parameter text⟩ of the macro definition. However, these tokens are not within the same curly-brace-group as the macro token whose macro arguments are gathered. As a result, TeX does not take this token sequence for the matching argument delimiter.
Taking such a sequence for a matching argument delimiter would imply the possibility of delimited macro arguments where explicit character tokens of category 1(begin group) or 2(end group) don't have matching counterparts (i.e. where curly braces don't have matching counterparts).
Therefore, proper curly-brace-nesting/proper balancing of explicit character tokens of category 1(begin group) with matching explicit character tokens of category 2(end group) is taken into account when searching an argument delimiter.
For instance, after defining
\def\macro#1\deli\miter{This is the argument: (#1)}
calling \macro as in
\macro PartOfArgument{\deli\miter}OtherPartOfArgument\deli\miter
yields that #1 from the ⟨parameter text⟩ of the definition with this call is considered a placeholder for
PartOfArgument{\deli\miter}OtherPartOfArgument
.
In cases of finding the matching argument delimiter, the tokens forming the argument delimiter are discarded. After finding and discarding the matching argument delimiter, TeX checks whether the entire set of tokens gathered for the macro argument so far (that set can be empty in case the delimiter is found immediately) is enclosed in a leading explicit character token of category 1(begin group), and a trailing explicit character token of category 2(end group). I.e., after finding and discarding the matching argument delimiter, TeX checks whether the entire set of tokens gathered for the macro argument so far is enclosed in curly brace tokens. If so, these two tokens are removed. (This can result in emptiness in case TeX only gathered an explicit character token of category 1(begin group) followed by an explicit character token of category 2(end group) when finding the delimiter—that is, if TeX gathered only a pair of matching curly braces before finding the delimiter.)
There are situations where you wish to avoid this removal of the outermost level of curly braces from delimited macro arguments. This can easily be achieved by putting a token in front of the actual argument which cannot accidentally/erroneously match the argument delimiter and at the end of the expansion-chain having this token removed.
E.g.:
(\detokenize in use, thus eTeX-extensions required this time ;-) )
\def\DoNothing{}%
\def\RemoveDot.{}%
\def\macro#1#2\delimited{\detokenize\expandafter{#1#2}}%
\tt
% Stripping of curly braces with the delimited argument will take place.
% #2 will be: 'Delimited Argument'
\macro{\DoNothing}{Delimited Argument}\delimited
% Due to the leading dot, stripping of curly braces with the delimited argument will not take place.
% #2 will be: '.{Delimited Argument}'
\macro{\RemoveDot}.{Delimited Argument}\delimited
\bye
When TeX starts gathering a delimited macro argument, it does not discard whatsoever token. It does not discard space tokens. When TeX starts gathering a delimited macro argument, anything that might be taken for a preceding space token is part of the set of tokens that forms the macro argument.
Let's exhibit these subtleties:
Assume ␣ denotes a space and \macro is defined as follows while LaTeX2e with eTeX-extensions is in use:
\def\macro#1#2\delimited{%
Undelimited argument: \expandafter\verb\scantokens{*|#1|};
Delimited argument: \expandafter\verb\scantokens{*|#2|}%
}%
When calling \macro{Undelimited}Delimited\delimited, at the delimited argument there is no ⟨space token⟩ and there are no surrounding curly braces, thus you get:
Undelimited argument: Undelimited; Delimited argument: Delimited
When calling \macro{Undelimited}␣Delimited\delimited, at the delimited argument there is a ⟨space token⟩ and there are no surrounding curly braces, thus you get:
Undelimited argument: Undelimited; Delimited argument: ␣Delimited
When calling \macro{Undelimited}{Delimited}\delimited, at the delimited argument there is no ⟨space token⟩ and there are surrounding curly braces—the surrounding curly braces are removed, thus you get:
Undelimited argument: Undelimited; Delimited argument: Delimited
When calling \macro{Undelimited}␣{Delimited}\delimited, at the delimited argument there is a ⟨space token⟩ and there are curly braces. This time they do not surround
the entire delimited argument because the ⟨space token⟩ is there too. Thus this time they are not removed:
Undelimited argument: Undelimited; Delimited argument: ␣{Delimited}
There is a subtle thing in TeX: #{-notation:
When via \def giving the definition of a macro, the ⟨replacement text⟩ must be enclosed in an explicit character token of category 1(begin group) and an explicit character token of category 2(end group). I.e., when via \def giving the definition of a macro, the ⟨replacement text⟩ must be enclosed in curly braces.
Therefore you can write:
\def\macro{replacement text}
{\tt \meaning\macro}
\bye
When via \def giving the definition of a macro, as a special case the last token of the ⟨parameter text⟩ can be a single character token of category 6(parameter), be it explicit, e.g., a hash (#6), or implicit. If this is the case, then the explicit character token of category 1(begin group) forming the left side of the enclosing of the ⟨replacement text⟩ is also considered the last token of the delimiter of the last macro argument. Usually, when gathering tokens that are to form a delimited macro argument, the tokens forming the argument-delimiter are discarded. But in this special case, that last token of the argument delimiter of the last macro argument is not just discarded but is discarded and re-inserted, which is the same as if it would be left in place, and after expanding the macro in question that token occurs right behind the replacement text.
In the TeXbook it is explained that you can use this #{-feature for having the last macro argument of some macro delimited by an opening curly brace that is left in place so that it can, e.g., serve as opening curly brace/as the left side of the enclosing of a subsequent enclosed undelimited macro argument which in turn is to be gathered when expanding another macro token which is to come into being in the course of the expansion-cascade.
Example:
\tolerance 9999
\emergencystretch 3em
\hfuzz 0pt \vfuzz \hfuzz
\parindent=0ex
%-------------------------------------------------------------------------------------------------
\let\ImplicitHash=#
\def\macroA#1#2\relax#{Argument1:(#1), Argument2:(#2) The meaning of the next token: \meaning}
\def\macroB#1#2\relax\ImplicitHash{Argument1:(#1), Argument2:(#2) The meaning of the next token: \meaning}
\def\Weird{Whatsoever}
\tt\frenchspacing
\meaning\macroA
\noindent\hrulefill\null
\macroA{Arg 1}{Arg 2}\relax{ The meaning of the next token: \meaning}
\noindent\hrulefill\null
\meaning\macroB
\noindent\hrulefill\null
\macroB{Arg 1}{Arg 2}\relax{ The meaning of the next token: \meaning}
\bye
Until TeX, version 3.141592653, January 2021, there was an edge thing
(It was brought to my attention by others in the usenet newsgroup comp.text.tex in 2009. See, e.g., the usenet-discussion "Is there any difference between { } and bgroup and egroup ?")
When via \def giving the definition of a macro, as a special case the last token of the ⟨parameter text⟩ can be a single character token of category 6(parameter), be it explicit, e.g., a hash (#6), or implicit. In that special case you are not bound to the left side of the enclosing of the ⟨replacement text⟩ being an explicit character token of category 1(begin group). In that special case the left side of the enclosing of the ⟨replacement text⟩ may also be an implicit character token of category 1(begin group). After expanding the macro in question, the token denoting the implicit character token of category 1(begin group) occurs right behind the replacement text in the same way in which an explicit character token of category 1(begin group) would occur right behind the replacement text if the left side of the enclosing of the replacement text of the macro had been an explicit character token of category 1(begin group).
This is the case even if the token, which at the time of defining the macro in question denotes the implicit character token of category 1(begin group) gets redefined after defining the macro in question.
Example:
\let\Weird={
\let\ImplicitHash=#
\def\macroA#1#2\relax#\Weird Argument1:(#1), Argument2:(#2)\string}
\def\macroB#1#2\relax\ImplicitHash\Weird Argument1:(#1), Argument2:(#2)\string}
\def\Weird{Whatsoever}
\tt
\meaning\macroA
\macroA{Arg 1}{Arg 2}\relax\Weird
\meaning\macroB
\macroB{Arg 1}{Arg 2}\relax\Weird
\bye
The most important things to keep in mind about TeX's gathering of macro arguments are:
TeX gathers macro arguments from the token stream one by one.
In case of gathering a delimited macro argument, TeX gathers tokens from the token stream until finding the delimiter. This implies that delimited macro arguments can be empty in case TeX has not yet gathered any token when finding the delimiter.
When gathering undelimited macro arguments, TeX discards preceding explicit space tokens.
When gathering delimited macro arguments, TeX does not discard explicit space tokens. With delimited macro arguments anything that might be taken for a preceding explicit space token is part of the set of tokens that forms the macro argument.
If present, one level of curly braces surrounding an entire macro argument, be it a delimited macro argument or an undelimited macro argument, is discarded/removed/stripped off.
With your definition
\def\foo#1#2\baz{\#1: -#1- \quad \#2: -#2-}
TeX at the time of expanding \foo in any case first gathers an undelimited macro argument and then gathers a delimited macro argument where \baz is the delimiter:
\foo hello world\baz
At the time of expanding \foo TeX gathers the tokens of the undelimited macro argument, discarding preceding space tokens if present, and then discarding curly braces surrounding the entire macro argument if present:
h
Then TeX obtains the delimited macro argument by gathering tokens until finding the delimiter \baz and then—if present—discarding curly braces surrounding the entire set of tokens gathered so far:
ello world
\foo supercalifragilistiexpialidocious\baz
At the time of expanding \foo TeX gathers the tokens of the undelimited macro argument, discarding preceding space tokens if present, and then discarding curly braces surrounding the entire macro argument if present:
s
Then TeX obtains the delimited macro argument by gathering tokens until finding the delimiter \baz and then—if present—discarding curly braces surrounding the entire set of tokens gathered so far:
upercalifragilistiexpialidocious
\foo a \baz
At the time of expanding \foo TeX gathers the tokens of the undelimited macro argument, discarding preceding space tokens if present, and then discarding curly braces surrounding the entire macro argument if present:
a
Then TeX obtains the delimited macro argument by gathering tokens until finding the delimiter \baz and then—if present—discarding curly braces surrounding the entire set of tokens gathered so far: As space tokens do get removed only when they are preceding undelimited macro arguments while this is a delimited macro argument, the ⟨space token⟩ between the a and the \baz-delimiter will be gathered as the macro argument.
\foo a\baz
At the time of expanding \foo TeX gathers the tokens of the undelimited macro argument, discarding preceding space tokens if present, and then discarding curly braces surrounding the entire macro argument if present:
a
Then TeX obtains the delimited macro argument by gathering tokens until finding the delimiter \baz and then—if present—discarding curly braces surrounding the entire set of tokens gathered so far: In this case the delimiter is found immediately, while not yet having gathered any tokens. Thus in this case the macro argument is empty.
