# fully expand toks and save for later (appendix with exact copies of tables from the main text)

I have a document with some complex tables, defined by a command called \Rule with named parameters (using the keyval package). The individual rows of the table are first stored into a token list defined via \newtoks because some of the named parameters correspond to multiple cells of the table and simultaneously must be enveloped inside an \ifcsname condition. I'd like to force \Rule to store a fully expanded copy of the resulting table (with the particular values for each named parameter) so that I can list all rules that appeared anywhere in the document in the appendix. But I was not able to force the token list to fully expand.

I compile with xetex and cannot change that.

EDIT: Each table corresponds to a single call of \Rule; the named parameters contain the information, and the definition of \Rule provides the formatting for each kind of named parameter (most are optional). I want exactly the same tables in main text and in the appendix: the output should be the same as if I manually copy-pasted all \Rule commands from the document into the appendix.

\documentclass{book}
\usepackage{xparse,keyval}

\newcounter{rule}
\renewcommand{\therule}[0]{G\arabic{rule}}

\makeatletter
\newtoks\@tabtoks
\newcommand*\resettabtoks{\@tabtoks{}}
\newcommand*\printtabtoks{\the\@tabtoks}

\edef\allrules{}
\edef\oldallrules{\allrules}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%      PARAMETERS       %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\define@key{myKeys}{nonnumbered}{\def\mm@nonnumbered{#1}}
\define@key{myKeys}{label}{\def\mm@label{#1}}
\define@key{myKeys}{FUNKTORY}{\def\mm@FUNKTORY{#1}}

%% one \ifcsname cannot directly generate more than a single table cell (problems with & and \\ characters)
%% so instead of spitting out the rows of the table immediately,
%% and printed when all parameters are read

\DeclareDocumentCommand{\Rule}{m}{
\begingroup%
\medskip\noindent
\begin{table}
\hline}
\ifcsname mm@nonnumbered\endcsname
\else      \refstepcounter{rule}
\fi
\ifcsname mm@OBLIG\endcsname        \addtabtoks{Obligatory          & \mm@OBLIG             \\}\fi
\end{tabular}}
\printtabtoks

%         %%%%%%%% BEFORE RESETTING \@tabtoks, I'D LIKE TO STORE ITS FULLY EXPANDED VALUE
%           \expandafter\long\expandafter\edef\expandafter\tmp{\expandafter\write\expandafter\the\@tabtoks}
%           \show\tmp
%           \show\allrules

\resettabtoks

\ifcsname mm@label\endcsname \label{\mm@label}\fi
\end{table}
\smallskip
\endgroup%
}
\makeatother

\begin{document}
\Rule{
nonnumbered = {yes},
FUNKTORY = {VALUE 1           & VALUE 2},
}

\Rule{
FUNKTORY = {X           & Y},
label = {rule:1},
}

\appendix
\allrules
\end{document}

• it isn't clear what you want to do, most tex commands including \begin{tabular} or \bf (which should never be used in latex) do not work by expansion so can not be completely expanded in an edef in any meaningful way. What is the actual top level requirement that you want to implement? – David Carlisle Jul 1 '19 at 7:06
• I think many things can be simplified, and expansion doesn't need to happen so early. table creates a float, why do you put \medskip\noindent before \begin{table} and \smallskip after \end{table}? Are you sure you want table environments? Without any caption? Do you want the labels to point to the rules in the main text or in the appendix? – frougon Jul 1 '19 at 8:47

It's not exactly clear what you want the final summary to look like, but this might get you started. As you see, the rules are all collected at the end.

What I do here is create a 2nd token register \globaltabtoks in parallel to the one employed by the OP, \@tabtoks. Then, I can add pieces-parts of the rules into the \globaltabtoks in the manner to construct a summary table. And I don't clear/reset the register between rules! So, for example, rather than setting a row between the title and the content, I merely employ a column separator.

The key is to expand the keys into their text before placing them into the token registers. Here I create \addglobaltabtoks{} to place literal text at the end of the \globaltabtoks register, and \xaddglobaltabtoks{} to expand-once the first token of the argument and then append it and the rest of the argument to \globaltabtoks.

In the case of \therule, I need more than a single expansion to get the desired number, so I used the new TeX primitive \expanded in the fashion of

\xaddglobaltabtoks{\expanded{\therule}&}


However, if your installation does not yet support \expanded, one could as an alternative use

\edef\tmp{\therule}


Here is the MWE:

\documentclass{book}
\usepackage[T1]{fontenc}
\usepackage{xparse,keyval}

\newcounter{rule}
\renewcommand{\therule}[0]{G\arabic{rule}}

\makeatletter
\newtoks\@tabtoks
\newtoks\globaltabtoks
\global\globaltabtoks\expandafter{\the\globaltabtoks#1}%
}
}
\newcommand*\resettabtoks{\@tabtoks{}}
\newcommand*\resetglobaltabtoks{\globaltabtoks{}}
\newcommand*\printtabtoks{\the\@tabtoks}

\edef\allrules{}
\edef\oldallrules{\allrules}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%      PARAMETERS       %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\define@key{myKeys}{nonnumbered}{\def\mm@nonnumbered{#1}}
\define@key{myKeys}{label}{\def\mm@label{#1}}
\define@key{myKeys}{FUNKTORY}{\def\mm@FUNKTORY{#1}}

%% one \ifcsname cannot directly generate more than a single table cell (problems with & and \\ characters)
%% so instead of spitting out the rows of the table immediately,
%% and printed when all parameters are read

\DeclareDocumentCommand{\Rule}{m}{
\begingroup%
\medskip\noindent
\begin{table}[ht]
\hline}
\ifcsname mm@nonnumbered\endcsname
\else      \refstepcounter{rule}
\fi
\ifcsname mm@OBLIG\endcsname        \addtabtoks{Obligatory          & \mm@OBLIG             \\}
\end{tabular}}
\printtabtoks

\resettabtoks

\ifcsname mm@label\endcsname \label{\mm@label}\fi
\end{table}
\smallskip
\endgroup%
}
\makeatother

\begin{document}
\Rule{
nonnumbered = {yes},
FUNKTORY = {VALUE 1           & VALUE 2},
}

\Rule{
FUNKTORY = {X           & Y},
label = {rule:1}
}

\Rule{
FUNKTORY = {P           & Q},
label = {rule:2},
}

\appendix SUMMARY

\noindent
\begin{tabular}{|l|p{5cm}|l|}
\hline
\the\globaltabtoks
\end{tabular}

\end{document}


• Nice idea to construct a summary table! I have a vague idea that the trick is that the outer command \xaddglobaltabtoks feeds an already expanded argument into the inner command \addglobaltabtoks which cares for maintaining the global value of \globaltabtoks, but if you could explain it a little further, it would be great. – Ansa211 Jul 1 '19 at 12:04
• If I place your code into a file and compile with xelatex, it gives me ! Undefined control sequence. <argument> \expanded (newline and indent)  {\therule }& l.85 } – Ansa211 Jul 1 '19 at 12:07
• @Ansa211 \expanded is a new TeX primitive that has just been introduced to the language within the last 6 months or so. It basically allows you to fully expand its argument in an expandable context. You have two choices: 1) upgrade your latex installation to the latest version, or 2) replace that code with \edef\tmp{\therule}\xaddglobaltabtoks{\tmp}&}. Note the presence of the \edef makes this latter code non-expandable. – Steven B. Segletes Jul 1 '19 at 12:32
• @Ansa211 I have added commentary to describe what I did. – Steven B. Segletes Jul 1 '19 at 12:41
• @Ansa211 Note, if the summary table gets too big to fit on a page, just change the summary tabular to the comparable syntax for longtable. – Steven B. Segletes Jul 1 '19 at 12:45

I think what you want is something like what follows. Macros can record many things. Most things. Verbatim material is tricky, but you don't have any here, so you can just record the high-level commands with their arguments in a LaTeX3 sequence and use it in the appendix to replay everything (precisely, in \recapallrules). I only had to introduce an indirection level in two places:

• For the \label command: there must be only one label with a given name, so I use \mm@set@rule@label in \typesetRule. \mm@set@rule@label sets a label when used in the main text, but does nothing when used from \recapallrules, i.e., in the appendix (the behavior is trivial to change).

• For the counter incrementation: for the same reason, \mm@increment@rule@ctr does \refstepcounter{rule} when used in the main text and \stepcounter{rule} when used from \recapallrules (we don't want two different references—places recorded with a \label command—for the same value of the rule counter).

I also simplified your code by removing all the \addtabtoks-like things, which appear not to be necessary at all (unless proven otherwise!).

\documentclass{book}
\usepackage{keyval}
\usepackage{xparse}

\newcounter{rule}
\renewcommand{\therule}[0]{G\arabic{rule}}

\makeatletter

\define@key{myKeys}{nonnumbered}{\def\mm@nonnumbered{#1}}
\define@key{myKeys}{label}{\def\mm@label{#1}}
\define@key{myKeys}{FUNKTORY}{\def\mm@FUNKTORY{#1}}

\newcommand*{\mm@normal@increment@rule@ctr}{\refstepcounter{rule}}
\newcommand*{\mm@appendix@increment@rule@ctr}{\stepcounter{rule}}
\let\mm@increment@rule@ctr=\mm@normal@increment@rule@ctr

\newcommand*{\mm@normal@rule@label@cmd}{%
\ifcsname mm@label\endcsname
\expandafter\label\expandafter{\mm@label}%
\fi
}
\newcommand*{\mm@appendix@rule@label@cmd}{}
\let\mm@set@rule@label=\mm@normal@rule@label@cmd

\NewDocumentCommand{\typesetRule}{m}{% don't forget the percent sign here!
\begingroup
\mm@set@rule@label
\begin{tabular}{|p{5cm}|l|}
\hline
\ifcsname mm@nonnumbered\endcsname
\else
\multicolumn{2}{|l|}{{%
\mm@increment@rule@ctr
\fi
\hline
\ifcsname mm@FUNKTORY\endcsname
\mm@FUNKTORY \\
\fi
\ifcsname mm@OBLIG\endcsname
Obligatory & \mm@OBLIG \\
\fi
\hline
\end{tabular}%
\endgroup
}

\ExplSyntaxOn                   % Use LaTeX3-style input syntax (see expl3.pdf)

\seq_new:N \g_mm_rules_seq

\NewDocumentCommand \Rule { m }
{
\seq_gput_right:Nn \g_mm_rules_seq { \typesetRule {#1} }
\medskip\noindent
\typesetRule {#1}
}

% Programming-level function
\cs_new_protected:Npn \mm_recap_all_rules:
{
\group_begin:               % limit effect of the \cs_set_eq:NN
\setcounter {rule} {0}      % reset the rule counter
% Special version of two commands for use here (we used different versions
% in the main text). Note that \cs_set_eq:NN is like \let.
\cs_set_eq:NN \mm@increment@rule@ctr \mm@appendix@increment@rule@ctr
\cs_set_eq:NN \mm@set@rule@label \mm@appendix@rule@label@cmd
% Use all elements we've recorded, separated using the provided separator
\seq_use:Nn \g_mm_rules_seq { \par \medskip \noindent } % separator
\group_end:
}

% User-level function
\NewDocumentCommand \recapallrules { }
{
\mm_recap_all_rules:
}

\ExplSyntaxOff                  % end of LaTeX3-style input syntax

\makeatother

\begin{document}

\Rule{
nonnumbered = {yes},
FUNKTORY = {VALUE 1           & VALUE 2},
}

\Rule{
FUNKTORY = {X           & Y},
label = {rule:1},
}

\Rule{
FUNKTORY = {Z           & T},
label = {rule:2},
}

\appendix

\chapter{All rules}

\recapallrules

\end{document}


On page 1:

Appendix:

I wouldn't mix “legacy” programming with expl3. With careful usage of variants, it's easy to control what gets expanded and what doesn't.

\documentclass{book}
\usepackage{xparse}

\newcounter{rule}
\renewcommand{\therule}[0]{G\arabic{rule}}

\ExplSyntaxOn

\NewDocumentCommand{\Rule}{m}
{
\group_begin:
\tl_clear:N \l__ansa_rule_body_tl
\keys_set:nn { ansa/rule } { #1 }
\__ansa_rule:
\tl_gput_right:NV \g__ansa_rule_all_tl \l__ansa_rule_body_tl
\tl_gput_right:Nn \g__ansa_rule_all_tl { \par\bigskip }
\group_end:
}
\NewDocumentCommand{\allrules}{}
{
\begin{center}
\cs_set_eq:NN \__ansa_rule_label:n \use_none:n
\tl_use:N \g__ansa_rule_all_tl
\end{center}
}

\tl_new:N \l__ansa_rule_body_tl
\tl_new:N \g__ansa_rule_all_tl

\keys_define:nn { ansa/rule }
{
nonnumbered .bool_set:N = \l__ansa_rule_nonnumbered_bool,
nonnumbered .default:n  = true,
nonnumbered .initial:n  = false,
label       .tl_set:N   = \l__ansa_rule_label_tl,
FUNKTORY    .tl_set:N   = \l__ansa_rule_funktory_tl,
OBLIG       .tl_set:N   = \l__ansa_rule_oblig_tl,
}

\cs_new_protected:Nn \__ansa_rule:
{
\bool_if:NTF \l__ansa_rule_nonnumbered_bool
{
}
{
\refstepcounter{rule}
\tl_if_empty:NF \l__ansa_rule_label_tl { \__ansa_rule_label:n {\l__ansa_rule_label_tl} }
}
\tl_if_empty:NF \l__ansa_rule_funktory_tl
{
}
\tl_if_empty:NF \l__ansa_rule_oblig_tl
{
}
\tl_use:N \l__ansa_rule_body_tl
}

\cs_new:Nn \__ansa_rule_label:n { \label{#1} }

{
\tl_put_right:Nn \l__ansa_rule_body_tl { #1 }
}

{
\__ansa_rule_add:n { \multicolumn{2}{|l|}{\bfseries#1\hfill Rule~#2} \\ }
}

{
}

\ExplSyntaxOff

\begin{document}

\Rule{
nonnumbered,