Why is & special outside of tabular environments?

Question

Consider the following m(non-)we:

\documentclass{standalone}
\begin{document}
Foo & Bar
\end{document}

The build will fail with

! Misplaced alignment tab character &.

while it will generate a perfectly fine PDF when removing the & character from the code.

Even plain TeX behaves like that and

Foo & Bar
\bye

yields the same error.

I have only ever seen & used in tabular and array environments, never outside of it. Why is that?

Couldn't one define e.g. \tabular and \endtabular so that & is only special inside the environment, and just a normal character outside of these environments?

Why does & outside of tabular and array environments have to break the document?

To rephrase the question: WTF is an alignment tab character and where do we usually put it?

Answer 1

Let's assume that the format or the user doesn't set the catcode of & (or any other symbol) globally to 4.

What would be the consequences assuming that you still want to have locally such a character (and not use commands like \nc for "new column" all the time)?

1. Every tabular like environment would have to make sure to set a char to be the alignment tab -- and they must agree which one to use. This would involve a lot of patching. This is naturally not an argument why it hasn't be done when the format was written.
2. It wouldn't be possible to use this character if the tabular is used in the argument of some command. E.g. you would run into problems when using tabulars in footers and headers, in addresses of letters, in author lists, in beamer frames. One could naturally define a command as a replacement, but users would have to understand when this command is needed.

3. It would be rather difficult to disable locally the alignment tab. E.g. currently you can easily type a tabular with lots of & like this:

   \documentclass{article}
   
   \begin{document}
   \begingroup
   \catcode`\&=12
   \catcode`!=4
   \begin{tabular}{ll}
   a & b ! c & d
   \end{tabular}
   \endgroup    
   \end{document}
   

   But if tabular resets the catcode of & it will no longer work. So you would need some hooks to allow such manipulations -- and again in every tabular like environment.

On the whole a lot things would get much more difficult for package writers and for users. It is easer to learn to type \&.

Answer 2

You ask:

Why is & special outside of tabular environments?

I think the question is not well-posed. I would say that & is special everywhere, not just outside tabular environments. The character & has category code 4 -- "alignment tab" -- in all TeX and LaTeX formats I've ever encountered. This catcode setting allows it to be used as the alignment tab character not only in tabular-like environments (e.g., tabular*, tabularx, longtable, etc), but also array environments, the deprecated eqnarray environment, some of the display math environments of the amsmath package, the IEEEeqnarray environment of the IEEEtrantools package, and probably quite a few additional environments.

The virtue of assigning catcode 4 to & by default is that all package writers can (and, frankly, should ) write their code assuming that & has catcode 4. Without this default, you'd have some package writers choosing & as the alignment tab character, whereas others might choose @ or who knows what else. Such an outcome would be distinctly inferior for most users, since they'd have to memorize which character, or characters, serve as the alignment tab function in which environments.

Conversely, if one does not want & to be special -- and doesn't want to modify its catcode -- one must input it as \&.