What can and can't occur as macro argument?

Question

The documentation of the environ package says

The advantage to using environments is that their contents is not treated as a macro argument, so there are less restrictions on what can exist inside, and the processing can be more efficient for long pieces of document text.

However, sometimes it might be useful to "scan" large amounts of code, parse them and "do nothing" unless some condition is satisfied. When doing so, it is important to know if this imposes limitations on the rest of the code.

Concretely, consider the following document:

\documentclass{minimal}

\long\def\everything#1end@of@everything{#1}
\def\test{test1}

\begin{document}
\everything
Hello World!

\test

%%% Some
%%% other
%%% code
end@of@everything
\end{document}

This works fine but what are the "dangers" of using such a macro? What can and can't occur in "Some other code"?

Two things that come to mind:

• anything defined with \outer should throw an error (see also When is it appropriate to use \outer?)

• there will probably be memory issues, but how bad are these, i.e. is there some limit to the length of a parameter?

---

EDIT: @Joseph Wright has pointed out nesting and (re-)tokenization issues (which is probably why the \collect@body commands of amsmath and environ work more subtly than my macro).

However, I'm still interested in the "efficiency" aspect. Taking this example from @Yiannis Lazarides, I find that

\documentclass{minimal}

\long\def\everything#1end@of@everything{#1}

\usepackage{lipsum}

\begin{document}
\everything

\newcount\n
\n=0
\def\message{I can count to }
   \loop
   \ifnum\n<37000
   \advance\n by1
   \message\number\n : \lipsum[1-2]    

   \repeat
end@of@everything
\end{document}

still works (producing an amazing 11563 pages of output :)), leading me to wonder if there are in fact any restrictions on the amount of data being passed as an argument.

Answer 1

Two classic limitations here are verbatim material and nesting.

If you grab material as a macro argument then TeX tokenizes it. Even with e-TeX (or writing to an external file) our ability to re-tokenize later is limited. Thus

\begin{foo}
  Some normal content
  \verb=%=
\end{foo}

can (typically) be handled quite happily, while

\foo
  \verb=%=
@endfoo

won't be.

In terms of nesting, using a delimited macro TeX will stop at the first match

\foo
  \foo
   stuff
  @endfoo
  morestuff
@endfoo

will therefore not act as expected while

\begin{foo}
  \begin{foo}
    stuff
  \end{foo}
  more stuff
\end{foo}

will.

(I've used LaTeX-like environments here, but you can of course define environments in plain TeX, ConTeXt, etc. with the same implications.)