# Need help with grouping and expansion control

I am trying to define macros, which take a delimited list and pass each element to a macro (that takes one argument). I have fully working versions of them that iterate over 1) all elements in a list, 2) the first n, (where n is given as a macro argument) but not adding empty ones if less than n, and 3) the first n, but adding empty elements if needed. They use grouping to make all definitions local but actually execute the macro after the group ends (which is essential). I've tested them with all sorts of nested macros that call them again with different delimiters and they seem to be robust. However, I am facing a very strange issue when trying to use them inside another macro which is supposed to take the first n elements of list (with empty ones at the end, if needed) and call a macro accepting n arguments. If I copy-paste the majority of the code from the 3rd version of the above mentioned macros, all is fine. If, however, I try calling this macro, asking it, for each list element, to call a locally defined macro which is supposed to append those elements to a (local) token list, the token list is being emptied after each iteration... I think the problem lies somewhere in the helper macros that append a token to list, as I show below, but I can't spot the cause! They work as expected in other cases. I must be missing something obvious and stupid, I apologize. Here are the relevant macros:

\documentclass{article}

\usepackage{etoolbox}
\makeatletter
\newtoks\tmp@toks

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% append token #1 to token list #2
\def\append@tok{\@ifstar\@append@tok\@append@tok@wrapped}
\def\@append@tok@wrapped#1#2{% wrapped in braces
#2=\expandafter{\the#2{#1}}%
}%
\def\@append@tok#1#2{% as is
#2=\expandafter{\the#2#1}%
}%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% iterate over a list, up to #1 many elements, adding empty ones if needed
\def\apply@macro@to@fixed@list#1#2#3#4{%
% #1: exactly that many elements; add empty ones as needed,
% #2: macro of one argument, #3: list delimiter, #4: list
\begingroup%
\def\next@el##1##2#3##3\@nil{%
% ##1: "counter", ##2: next element, ##3: rest of list
\append@tok{#2{##2}}\tmp@toks%
\ifnum##1<#1\relax%
\ifstrempty{##3}{%
% call it with an empty element (delimiter)
\expandafter\next@el\number\numexpr ##1+1\relax #3\@nil%
}{%
\expandafter\next@el\number\numexpr ##1+1\relax ##3\@nil%
}%
\fi%
}%
\tmp@toks={}%
\next@el{1}#4#3\@nil%
\expandafter\endgroup\the\tmp@toks%
}
% call a macro with the first #1 elements of list
% THIS DOES NOT WORK
\def\apply@macro@to@split@list@A#1#2#3#4{%
% #1: exactly that many elements; add empty ones as needed,
% #2: macro of #1 number of argument, #3: list delimiter, #4: list
% for some reason this did not work... after exiting from \apply@macro@...
% the token list was empty (but was correctly appended to inside \save@tok))
\begingroup%
\def\save@tok##1{%
\append@tok{##1}\tmp@toks% does not append, simply reassigns
% not really sure how to display the content without using \meaning and a buffer macro
\expandafter\def\expandafter\@tmp\expandafter{\the\tmp@toks}%
token list is \texttt{\meaning\@tmp}\par%
}%
% idea is to put the macro to be called at the front of the token list
% then append its arguments; for debugging purposes, do not do that
%  \tmp@toks={#2}%
\tmp@toks={}%
\apply@macro@to@fixed@list{#1}\save@tok{#3}{#4}%
exiting group, token list is \the\tmp@toks\par%
% and execute the macro outside the group
%  \expandafter\endgroup\the\tmp@toks%
\endgroup%
}
\def\apply@macro@to@split@list@B#1#2#3#4{%
\begingroup%
\def\save@tok##1{%
\append@tok*{{##1}}\tmp@toks% does not append either, simply reassigns
\expandafter\def\expandafter\@tmp\expandafter{\the\tmp@toks}
token list is \texttt{\meaning\@tmp}\par%
}%
\tmp@toks={}%
\apply@macro@to@fixed@list{#1}\save@tok{#3}{#4}%
exiting group, token list is \the\tmp@toks\par%
\endgroup%
}

\begin{document}
% to see that nested calls to \apply@macro@to@fixed@list work:
\def\foo#1{%
% use a different delimiter
\apply@macro@to@fixed@list{3}\bar:{#1a:#1b:#1c:#1d}%
}
\def\bar#1{element is: #1\par}
\apply@macro@to@fixed@list{3}\foo,{a,b,c,d}%
%
\def\foo#1#2{FOO: #1, #2;\par}
% using unstarred \@append@tok, i.e. wraps it in braces
\apply@macro@to@split@list@A{2}\foo:{a:}
% using starred \@append@tok, I wrap it in braces before passing it to \append@tok
\apply@macro@to@split@list@B{2}\foo:{a:}
% now I redefine \append@tok to be exactly the same as its starred version
\def\append@tok#1#2{% as is
#2=\expandafter{\the#2#1}%
}%
% and pass it a wrapped token, as before
\def\apply@macro@to@split@list@C#1#2#3#4{%
\begingroup%
\def\save@tok##1{%
\append@tok{{##1}}\tmp@toks%
\expandafter\def\expandafter\@tmp\expandafter{\the\tmp@toks}
token list is \texttt{\meaning\@tmp}\par%
}%
\tmp@toks={}%
\apply@macro@to@fixed@list{#1}\save@tok{#3}{#4}%
exiting group, token list is \the\tmp@toks\par%
\endgroup%
}
% and it works
\apply@macro@to@split@list@C{2}\foo:{a:}
% it also works if I append the token without using a helper macro
\def\apply@macro@to@split@list@D#1#2#3#4{%
\begingroup%
\def\save@tok##1{%
\tmp@toks=\expandafter{\the\tmp@toks {##1}}%
\expandafter\def\expandafter\@tmp\expandafter{\the\tmp@toks}
token list is \texttt{\meaning\@tmp}\par%
}%
\tmp@toks={}%
\apply@macro@to@fixed@list{#1}\save@tok{#3}{#4}%
exiting group, token list is \the\tmp@toks\par%
\endgroup%
}
\apply@macro@to@split@list@D{2}\foo:{a:}
\end{document}


Also, I am wondering, is my general approach and use of techniques suitable?

• I'm afraid that dumping so much code is not the best thing for getting sensible answer. – egreg Apr 9 '16 at 15:33
• @egreg I tried to make it as dense as possible... I can't pinpoint the problem, so that really is my "minimal" (not) working example – Aayla Secura Apr 10 '16 at 0:23