Appending active and expansion sensitive characters to a macro

Question

This question comes up pretty often when people want to build tables with empty columns or similarly building up some active characters inside a macro hoping to expand them at the right time right place.

Obviously, things don't work out as expected thanks to the intimate bowel movements of TeX (since we have mouth and stomach already). I've been asked this question in many different forms a lot of times and I think it has enough potential to be a duplicate source, hence the question. I also have a hacky solution but I think our wizards can make things more structured.

Consider the following toy example which has enough germs in it:

TikZ has a \matrix macro that acts like a tabular environment, that is, it looks for a row and a column separator and forms a graphical object. It doesn't have to be related to TikZ (see below)

\documentclass[tikz]{standalone}
\usetikzlibrary{matrix}
\begin{document}
\tikz\matrix[execute at empty cell=\node{$\bullet$};]{\themacrotobedefined};
\end{document}

or you can consider

\documentclass{standalone}
\begin{document}
\begin{tabular}{cccc}\themacrotobedefined\end{tabular}
\end{document}

The macro to be defined, \themacrotobedefined should hold the character stream

&&&\\&&&\\&&&\\

such that we have four columns, three rows (numbers are random choose what you wish). The question is as general as you wish, you can include math chars etc. But just to make it a bit more problematic, please use the following

\foreach\x{1,2,3}{
  \foreach\y{1,2,3}{
     ...
  }
}

so that the macro is build up sequentially at each spin (again foreach is optional do/while is also OK). The main goal is how to append sensitive items to macros without expanding them prematurely. plain TeX/LaTeX/ConteXt bring it on.

We have pretty high number of questions which touch this issue tangentially but never head-on so I think it is not a duplicate (unless we find it).

Answer 1

This is perhaps the simplest method, note that the cell macro itself is not expanded at all but the loop variables are expanded while constructing the macro. I have to use \& rather than & because of some weirdness in the tikz matrix code, with tabular you could use & rather than \noexpand\&.

\documentclass[tikz]{standalone}
\usetikzlibrary{matrix}


\begin{document}

\def\mycell#1#2{\node{${#1}^2-{#2}^2=\the\numexpr(#1+#2)*(#1-#2)\relax$};}

\def\themacrotobedefined{}
\foreach\x in {1,2,3}{%
  \foreach\y in {1,2,3}{%
  \xdef\themacrotobedefined{\expandafter\unexpanded\expandafter{\themacrotobedefined}%
    \noexpand\mycell{\x}{\y}%
   \ifnum\y=3 \noexpand\\\else\noexpand\&\fi}%
  }%
}

\texttt{\meaning\themacrotobedefined}
\tikz\matrix[ampersand replacement=\&,execute at empty cell=\node{$\bullet$};]{\themacrotobedefined};
\end{document}

---

The above builds uo \themacrotobedefined incrementally redefining it multiple times adding one cell each iteration. This is necessary as \foreach is not expandable internally it uses assignments to iterate the loop.

If you use an expandable loop mapping construct such as the one below, you can define the macro in a single definition:

\documentclass[tikz]{standalone}
\usetikzlibrary{matrix}

\def\myloop#1#2#3{%
  #3{#1}%
  \ifnum#1=#2 \expandafter\gobblesix\fi
  \expandafter\myloop\expandafter{\the\numexpr#1+1\relax}{#2}{#3}}
\def\gobblesix#1#2#3#4#5#6{}

\begin{document}


\def\mycell#1#2{\noexpand\node{${#1}^2-{#2}^2=\the\numexpr(#1+#2)*(#1-#2)\relax$};%
\ifnum#2=3 \noexpand\\\else\noexpand\&\fi
}
\def\zmycell#1{\myloop{1}{3}{\mycell{#1}}}


\edef\themacrotobedefined{%
  \myloop{1}{3}{\zmycell}%
}


\texttt{\meaning\themacrotobedefined}
\tikz\matrix[ampersand replacement=\&,execute at empty cell=\node{$\bullet$};]{\themacrotobedefined};
\end{document}

Answer 2

I added the solution similar to the @David's solution, but using the macros \gaddto and \xaddto. The \gaddto adds the unexpanded parameter to the macro and the \xaddto adds the expanded parameter to the macro. The usage (in the loop) is less tricky then using \expandafter\unexpanded\expandafter etc.:

\long\def\gaddto#1#2{\expandafter\gdef\expandafter#1\expandafter{#1#2}}
\long\def\xaddto#1#2{\edef\tmp{#2}\expandafter\gaddto\expandafter#1\expandafter{\tmp}}

\def\tobedefined{}
\foreach\x in {1,2,3}{%
  \foreach\y in {1,2,3}{%
    \gaddto\tobedefined{\mycell}
    \xaddto\tobedefined{{\x}{\y}}
    \ifnum\y=3 \gaddto\tobedefined{\\}\else\gaddto\tobedefined{\&}\fi
}}

Answer 3

Your question seems to be about preparing some macro by adding material to it which may be or not be expanded. This looks like a general question, but if one focuses on \foreach then it becomes immediately another question which is: how can one overcome the fact that \foreach does each item in a group ? and also how does one convert the \x, or \y loop variables into the values they hold ? This is addressed already in the provided answers.

Sometimes one wished one would not have to construct such a macro in advance and rather insert it directly in the structure at hand like a tabular or a \tikz\matrix. In the favorable cases like tabular one knows that with expandable loops one can do the thing, however one has mainly to be careful about when TeX sees the & (the \cr are hidden in the LaTeX \\), and when it is the right time to do some assignment safely. And with \numexpr construct one can do things without any assignment. And even if one does assignments, one does not have to make them global (as is demonstrated by \xintFor).

But these are the favorable cases: in the less favorable cases you depend on how the surrounding environment is going to behave: will it scan its contents? will it expand it more than one time ? even in contexts which look like they could accept stuff working purely by expansion, there may be surprises. For example the coordinates of TikZ/PGF seems to accept expanding only up to 100 times its contents, not more.

Ok, so the safest is always to build a macro in advance but it is not sufficiently well-know that this is NOT necessary for a tabular (Questions on this topic have a tendency to be quickly closed as duplicates).

Then, as you already got answer about how to use \foreach, I provide a somewhat off-topic answer of using another loop which does not do each item in a group. Although not an expandable loop it works if used directly within a tabular. For \tikz\matrix however, I must fall back on defining the \Macro before, and as observed in David's answer it appears that for \tikz\matrix one has to prepare the macro with \& not with &.

\documentclass {article}

\usepackage{tikz}
\usetikzlibrary{matrix}

\usepackage{xinttools}

\makeatletter
\let\firstofone\@firstofone
\let\gobble\@gobble
\makeatother

\begin{document}

\begin{tabular}{*5{|c}|}
\hline
   \xintFor #1 in {1, 2, 3, 4, 5}\do
  {%
   \xintFor #2 in {1, 2, 3, 4, 5}\do 
    {${#1}^2-{#2}^2=\the\numexpr(#1+#2)*(#1-#2)$
     \ifnum#2=5 \expandafter\gobble\else\expandafter\firstofone\fi 
     {&}%
     }% 
   \\\hline
   }
%\hline
\end{tabular}


\newcommand*\mycell [2]
   {\node{${#1}^2-{#2}^2=\the\numexpr(#1+#2)*(#1-#2)\relax$};}

\def\Macro{}

\xintFor #1 in {1, 2, 3, 4, 5}\do
  {%
   \xintFor #2 in {1, 2, 3, 4, 5}\do 
   {%
      \odef\Macro {\Macro \mycell {#1}{#2}}%
      \ifnum #2<5
        \odef\Macro {\Macro \&}% use of \& in place of & for TikZ
      \fi 
   }%
   \odef\Macro {\Macro \\}%
}%

\ttfamily
\meaning\Macro

\begin{tikzpicture}
\matrix[ampersand replacement = \&, execute at empty cell=\node{$\bullet$};]
{\Macro };
\end{tikzpicture}
\end{document}

In the code above \odef\Macro{<stuff>} defines \Macro to be the once-expanded <stuff>.