Collect contents of an environment (that contains verbatim content)

There is some useful background info that should hopefully make clear exactly what I am doing, as well as the setting this is happening in which is relevant. I'll include the problem-specific info after the ----- line.

This is for an open source project for education materials called Ximera. I am editing the code for use at a different university, and pushing those changes as they are functional up to the original project. You can find all the edited code for the cls file here (any and all incidental code is also in that github repo).

The project uses a static compile order which gives me somewhat limited options in terms of more creative solutions (for example, I would use sage for some of the desired content, but the compiler is static and won't compile the sage). The compiler isn't customized for the tex code, although it does do some magic with mathjax and some other parts that I honestly am not sure how it works.

A lot of this code, being an open source project, is somewhat cobbled together. I've been in the process of streamlining, cleaning, and commenting, as much of the code relevant to this part of the project as I can, so it should be considerably more readable than one may expect however.

Now, my actual problem is in implementing the shuffle environment features in conjunction with the "python" code environment. You can search down to the shuffle environment code in the above link (around line 860) to get the full code for shuffle which is (relatively) self contained and you can find the python/code environment (around line 750).

Immediately after the "shuffle" environment itself is all the code for the problem environment types (there are several, but the only difference is the displayed name, which should be clear from the implementation code).

My thought was to use the Environ command to collect the contents of any of the problem contents, assign it to a command that is created based on the shuffle and then problem number. Then when the shuffle environment closes, we iterate over the questions to display them in a random order. Random numbers provided by the pgfplots package, and a custom command I wrote to generate and permute a vector of numbers, saved in commands and counters.

As the code is written in the link, shuffle works great with the various problem environments. However, apparently environ package doesn't play well with verbatim environments. And since the code/python environments are verbatim environments, if we include one of these in a problem, everything blows up. So, here is a full working/nonworking example using the project code from the link above.

%\documentclass[handout]{ximera}
\documentclass[]{ximera}

\begin{document}

\begin{shuffle}[5]
\begin{exploration}% This is a basic one example that works.

Compute the following derivative:
%\expandafter\input{\file@loc Derivatives/2311-Compute-Derivative-0001.HELP.tex}
$\dfrac{d}{dx}\left({x^{2} - 8 \, x + 16}\right)=\answer{2 \, x - 8}$

\end{exploration}

\begin{problem}% This example fails due to python
This is the second problem! The answer is $\answer{6}$
\begin{hint}
test
\end{hint}

%\begin{python}% python throws a fancyvrb error
%Test
%\end{python}
\end{problem}

\begin{question}% Display mode works as an environment
This is the third problem! I need to include some other code, so here;
$3x + 1 = 5$
What is $x$? $\answer{\frac{4}{3}}$
\end{question}

\begin{exercise}
This is the fourth problem!
\begin{theorem}% Theorem environment works
The Intermediate Value Theorem is never done right by students. What is the point of this theorem?
\end{theorem}
$\answer{Nothing}$

\begin{exploration}% This doesn't appear, probably because of the approach
This just doesn't appear
\end{exploration}

\end{exercise}

\begin{problem}% This example fails.
This is the fifth problem! Lets try some nested content

%Verbatim env throw errors
%\begin{verbatim}
%Here is a hint!
%\end{verbatim}

\end{problem}

\end{shuffle}

\end{document}


The above has examples of the things that work and don't work. The verbatim and python environments straight up fail (I assume for the same reason, since python is a verbatim env created by fancyvrb). The environ package is useful for a lot of other things, so if anything I'd like to fix the verbatim side of the problem if that's possible.

As a side note, I also can't do nesting problems yet. I'm pretty sure that's just because of how shuffle is functioning and that's not too high on my list right now. It would be nice to get nesting to work, but not necessary.

• If all of the environment body is to be read verbatim then things may be possible, if you want to grab a mixture of verbatim and non verbatim material then in general it's not really possible you probably need to give more details of the actual use case – David Carlisle Jun 6 '17 at 21:51
• Thanks for the reply; In general I want to have a sub-environment like \begin{codeblock} where the stuff in that environment gets saved verbatim, and then have the ability to have that environment inside the problem environment. Thus the problem environment would execute all code contained within, with the understanding that "executing" the codeblock environment, means saving that chunk verbatim. Alternatively, if I can have a conditional with a renew environment type solution that would work too. But I can't find a way for the outer environment to detect there is a codeblock inside first. – Jason Jun 7 '17 at 0:07

You can easily implement an environment that reads its body verbatimized and then within a macro definition wraps things into \scantokens.

When carrying out the macro, \scantokens will get carried out, and things will be re-tokenized not under verbatim-catcode-régime but under normal catcode-régime which means that this time commands like \verb or environments like verbatim and other verbatim-like environments, e.g., code-listings, will get carried out.

The following example provides an environment Problem.

If you provide a name in the optional argument of that environment, you can afterwards restate the content of the environment by saying \RestateProblem{<Name>}.

It also works in cases where the Problem-environment contains things like \verb or verbatim-environments.

Problem-environments cannot be nested as the entire bodies of these environments are treated as macro-arguments that are delimited by the phrase \end{Problem} (also tokenized in verbatimized fashion).

It is also possible to create a mechanism similar to \label..\ref in order to have things written to .aux-file in verbatimized-fashion, providing the feature of both forward- and backward-referencing in the text.

If you use such a restatement-thingie in conjunction with macros like \section or environments like \theorem where counters get stepped, you might get unwanted results as each occurrence of a restatement would cause the respective counters to be stepped...

In case of forward-referencing, you'd need to ensure that all counters, control-sequences are already defined at the time when forward-reference takes place...

In case of backward-referencing, you'd need to ensure that all counters, control-sequences are not redefined in ways that cause trouble at the time when referencing takes place...

Be aware that I am not yet elaborating on this in detail as this kind of things requires a lot of cumbersome and time-consuming tuning according to actual/specific needs.

The risk of spending a lot of time and efforts on something that needs to be rewritten completely during the course of adjusting it to the actual needs of the questioner is high.

Therefore I suggest you to specify your needs in as much detail as possible.

Don't take your issue for a matter of showing modesty.

Instead give every information/request/requirement that is useful for providing help in an efficient way.

\documentclass{article}
\makeatletter
%%
%%  Snippet written by Ulrich Diez on June 8, 2017
%%
%%----------------------------------------------------------------------
%% Check whether argument is empty:
%%......................................................................
%% \UD@CheckWhetherNull{<Argument which is to be checked>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is empty>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is not empty>}%
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
%%
%% (\romannumeral expansion was introduced in order to overcome the
%% concerns and worries about improperly balanced \if..\else..\fi constructs.)
\newcommand\UD@firstoftwo[2]{#1}%
\newcommand\UD@secondoftwo[2]{#2}%
\newcommand\UD@exchange[2]{#2#1}%
\newcommand\UD@CheckWhetherNull[1]{%
\romannumeral0\expandafter\UD@secondoftwo\string{\expandafter
\UD@secondoftwo\expandafter{\expandafter{\string#1}\expandafter
\UD@secondoftwo\string}\expandafter\UD@firstoftwo\expandafter{\expandafter
\UD@secondoftwo\string}\expandafter\expandafter\UD@firstoftwo{ }{}%
\UD@secondoftwo}{\expandafter\expandafter\UD@firstoftwo{ }{}\UD@firstoftwo}%
}%
%%----------------------------------------------------------------------
%% Put a control sequence token in place instead of the string denoting
%% its name:
%%......................................................................
%%  \Name<emptiness or tokens other than braces>{<Name of Control Sequence>}
%%
%% yields:
%%
%%  <emptiness or tokens other than braces>\Controlsequence
%%
%% E.g.,
%%
%%   \Name foo{bar} -> foo\bar
%%   \Name{bar} -> \bar
%%   \Name\newcommand*{wEirdName}[1]{Arg 1: (#1)}
%%       -> \newcommand*\wEirdName[1]{Arg 1: (#1)}
%%
\newcommand\Name{}\long\def\Name#1#{\romannumeral\UD@name{#1}}%
\newcommand\UD@name[2]{%
\expandafter\UD@exchange\expandafter{\csname#2\endcsname}{0 #1}%
}%
%%----------------------------------------------------------------------
%% Implement generic stuff for reading arguments "verbatim":
%%......................................................................
%%
% Syntax of \UDcollectOneVerbArg
%
%   \UDcollectOneVerbArg{<^^M-replacement>}{<mandatory>}<delimiter of verbatim arg><verbatim arg><delimiter of verbatim arg>
%   yields:
%   <mandatory>{<verbatim arg>}
%
% with each character ^^M (usually=\endline-char) in <verbatim arg>
% replaced by token-sequence <^^M-replacement>.
%
% If reading <^^M-replacement> and <mandatory> from input is necessary,
% they will be read under unchanged catcode regime.
%
% The <verbatim arg> is also mandatory.
% It will be read under verbatim-catcode-conditions.
% There must be a leading character in front of it.
%
% Empty-lines will not be ignored.
%
% <delimiter of verbatim arg> will be read under verbatim-catcode-conditions
% if present.
%
%
\begingroup
\catcode\^^M=12 %
\UD@firstoftwo{%
\endgroup%
\newcommand\UDEndlreplace[2]{\romannumeral0\@UDEndlreplace{#2}#1^^M\relax{}}%
\newcommand*\@UDEndlreplace{}%
\long\def\@UDEndlreplace#1#2^^M#3\relax#4#5{%
\UD@CheckWhetherNull{#3}%
{ #5{#4#2}}{\@UDEndlreplace{#1}#3\relax{#4#2#1}{#5}}%
}%
}{}%
\newcommand\UDcollectOneVerbArg{\@UDOneVerbArg{\@UDcollectOneVerbArg}}%
\newcommand\@UDOneVerbArg[3]{%
\@bsphack
\begingroup
\let\do\@makeother\dospecials
\catcode\ =10 %
\@@UDOneVerbArg{#1}{#2}{#3}%
}%
\newcommand\@@UDOneVerbArg[4]{%
\do\ %
\catcode\^^M=12 %
\long\def\@tempb##1#4{%
\def\@tempb{##1}%
\expandafter\UDEndlreplace\expandafter{\@tempb}{#2}{\def\@tempb}%
\expandafter#1\expandafter{\@tempb}{#3}%
}%
\@tempb
}%
\newcommand\@UDcollectOneVerbArg[2]{%
\endgroup
\@esphack
#2{#1}%
}%
% Implementation of generic stuff for reading arguments "verbatim" done.
%%
%%
%%----------------------------------------------------------------------
%% Implement generic stuff for defining Problem-like environments:
%%......................................................................
% #1 - Name of environment
}%
% ##1 - \end{
% ##2 - }
% ####1 - <Name of environment>
\Name\newcommand{Restate####1}[1]{%
\Name{####1@########1}%
}%
\Name\newcommand{process####1}[2]{%
\UD@CheckWhetherNull{########1}{}{%
\Name\newcommand{####1@########1}{%
\scantokens{\csname UD@firstoftwo\endcsname{}{}########2\begingroup\catcode\X=14 X}\endgroup
}%
\Name\Name\global\let{####1@########1}={####1@########1}%
}%
\scantokens{\csname UD@firstoftwo\endcsname{}{}########2\begingroup\catcode\X=14 X}\endgroup
\end{####1}%
}%
\newenvironment{####1}[1][]{%
\UDcollectOneVerbArg{^^J}{\Name{process####1}{########1}}{##1####1##2}%
}{%
}%
}%
}%
}%
%%
%% Generic stuff for defining Problem-like environments done.
%%
\makeatother

\begin{document}

\fbox{Problem stated the first time:}

\begin{Problem}[First Problem]%
This is normal text.
\verb|This is from the verb command.|
\verb*|This is from the verb* command.|
This is normal text.
\begin{verbatim}
This is from the verbatim environment:
&%{}§_"´~
\end{verbatim}
\end{Problem}

\fbox{Problem restated:}

\RestateProblem{First Problem}

\fbox{Problem restated once more:}

\RestateProblem{First Problem}

\end{document}

• This is a dense answer that may work. I'm reading through it now, but I added a lot more detail and a link to the full code that's actually in use if you want to look at it. Thank you for your time and help Ulrich. – Jason Jun 10 '17 at 18:35
• The solution not found in TexLive 2018 (update)...saludos – Pablo González L May 23 '18 at 2:20

Did I get right what I read within the .cls-file?:

You wish to have an environment shuffle wherein specific other environments can be used for writing/saving portions of text. The saved portions of text shall be placed within the output-file/.pdf-file in random order when it comes to processing the end of the shuffle-environment.

If I got you right, a few questions/issues arise:

1. What to do about things within the environment shuffle that are not placed inside such specific other environments?

[ I see two possibilities:

1. Process these things normally.
2. Try to have these things ignored completely. They probably could be processed in the same way the environment tikzpicture for drawing images from the package TikZ  processes things, e.g., using a nullfont etc ensuring that no visible output gets produced. Or have things processed like in xcomment.sty where one can specify which environments to process while ignoring any other piece of code. ]
2. How to handle the case of nesting such specific other environments?

3. How to handle the case of the bodies of such specific other environments containing unbalanced things, e.g., unbalanced opening braces, unbalanced closing braces, unbalanced \if..\else..\fi, unbalanced \begingroup, unbalanced \endgroup, unbalanced \bgroup, unbalanced \egroup, unbalanced \begin{whatsoever environment}, unbalanced \end{whatsoever environment}, unbalanced \csname, unbalanced \endcsname, ...

4. If portions of text shall be placed in random order this means producing random numbers. (As far as I know, the package pgf comes along with a random number generator.)

I saw that you implemented some variant of the Fisher Yates Shuffle algorithm which implies needing a set of random numbers whose cardinality is smaller by 1 than the number of portions of text that are to be shuffled randomly.

[ Instead—I mention this rather as a moot point—you can also play around applying, e.g., the Lehmer Code ( <https://en.wikipedia.org/wiki/Lehmer_code> ) which is used for bijectively mapping to natural numbers in the range from 0 to (n! - 1) all permutations of a set of n different elements bijectively mappable to nominal numbers within range from 0 to (n - 1): For randomly choosing a permutation of k different elements (0<k ≤ n) from n such elements, choose a random number R  within the range from 0 to (n!/((n - k )! ) - 1), let the number S  be R·((n - k )! ), apply the Lehmer Decoding Algorithm to S  in order to obtain that permutation of n elements that by the Lehmer Code is mapped to S , and from that permutation keep only the first/leading k elements.
This would require only one random number at the cost of doing both a lot of arithmetic, e.g., for calculating the kth falling factorial of n, and Lehmer-decoding.
The arithmetic could probably be done using Heiko Oberdiek's package bigintcalc, but I suppose such a thing might turn out very time consuming at LaTeX-compilation-time. ]

Seems, in any case you need whatsoever set of random numbers.

The crucial point about LaTeX is:

Often it is required to compile the source code of a LaTeX document several times until \label-\ref-references or references to the bibliography and many other things match out correctly. If values of random variables get created anew within each LaTeX-compilation, you might get a never ending compilation-story as within each compilation the values of random variables, and thus the text connected to these random variables(!), might change while consecutive LaTeX-compilations are needed for making references match out correctly while during these consecutive compilations values of random variables and thus the text connected to these random variables might change again, again raising the need for consecutive LaTeX-compilations for making references match out correctly, again leading to changing the values of random variables, again leading to changing the text connected to these random variables ...

Therefore you might need some mechanism/system for saving values of random variables created during the first LaTeX-compilation for usage during consecutive LaTeX compilations. Probably something derived from the \label-\ref-mechanism so you can have (La)TeX try to get the value of a random variable from a label and if that fails create it via a random generator and in any case save it within a label for consecutive LaTeX-compilations... (The \label-\ref-mechanism provides facilities for warning/informing the user about the need of re-running LaTeX in case something changed.)

With such a mechanism/system you need naming conventions for these labels that allow the automatic creation of unique/unambiguous names for labels. Probably something based on counting both the shuffle-environments and counting the above-mentioned specific environments that may occur within the shuffle-environments...

• I happen to be reviewing this problem when you posted. I'd be happy to chat if/when you are available. I looked through your previous answer and I confess that it is well beyond my own technical latex skill. I have already considered most points raised in this post, but answering them in a comment isn't practical, so an interactive discussion may be a better option. I don't readily know how to do so in stackexchange however. – Jason Jun 13 '17 at 21:27
• As a short version, both for yourself and any others reading this, You have the right premise from the cls file. The shuffle environment is intended to only be an environment to shell the problem style environments. Anything not in a problem style environment would (ideally) be ignored/not printed. Moreover, each problem-style environment is a shell containing whatever contents in it (including other environments) and shifted around by shuffle. But shuffle only applies to the first tier problem-env. So any nested problems would travel with the outermost problem env. – Jason Jun 13 '17 at 21:43
• I have sent the email, thanks for the info. Once we get something working it may be worth posting it as a formal answer to get nested (possibly verbatim) environments working nicely (it seems to be a recurrent problem). So I'll hold off on "accepting" an answer until we decide if it's worth posting an answer on here. – Jason Jun 13 '17 at 22:25