What is the maximum limit of elements in a list (expl3/xparse)?

Question

I have a query with the maximum number of items that can be saved in an expl3 list. Thanks to the response that @egreg gave me in Avoid passing an empty [optional argument] an environment (with verbatim content) using xparse? I was able to finish the implementation of an environment capable of saving valid LaTeX code (including verbatim) in expl3lists ... something like a "ConTextbuffers".

Let me explain a bit, I have many question/answer files in which I have abused the use of filecontents to record temporary files and included them in environment based on environ (which did not support verbatim) and then use them in different parts of the documents.

With the new version of filecontentsdef it is not necessary to write the external files (included verbatim) to have them in a \macro, thus the things and gathering ideas I have managed to create a scontents environment which encapsulates \filecontentsdefmacro, what run and save in a seq list, which I can access with \getstored[index]{seq name}, the environment and command Scontents works perfectly.

My question is this, the environment expands the content of \macro and stores in sequence, the command stores direct in sequence (same or other), at some point the seq list will have a limit. What is the limit on the number of items I can save?

I have tested some files, but, I convert all my old files using a script to pass them to the new scontents environment.

I leave here the example code using l3seq

\documentclass{article}
\usepackage{filecontentsdef}[2019/04/20]
\usepackage{xparse}
\usepackage{pgffor,fvextra} % only for example
\pagestyle{empty}           % only for example
\setlength{\parindent}{0pt} % only for example
\ExplSyntaxOn
\tl_new:c { l_scontents_macro_tmp_tl }

\keys_define:nn { scontents }
  {
    save-env  .tl_set:N   = \l_scontents_name_seq_tl,
    save-env  .initial:n  = contents,
    show-env  .bool_set:N = \l_scontents_show_env_bool,
    show-env  .initial:n  = false,
    save-cmd  .tl_set:N   = \l_scontents_name_seq_cmd_tl,
    save-cmd  .initial:n  = contents,
    show-cmd  .bool_set:N = \l_scontents_show_cmd_bool,
    show-cmd  .initial:n  = false
  }

\cs_new_protected:Npn \_scontents_append_contents:nn #1#2
  {
    \seq_if_exist:cF { g_scontents_seq_name_#1_seq }
      {
        \seq_new:c { g_scontents_seq_name_#1_seq }
      }
    \seq_gput_right:cn { g_scontents_seq_name_#1_seq } {#2}
  }

\cs_new_protected:Npn \_scontents_getfrom_seq:nn #1#2
  {
    \seq_item:cn { g_scontents_seq_name_#2_seq } {#1}
  }

\ProvideExpandableDocumentCommand{ \countseqsc }{ m }
  {
    \seq_count:c { g_scontents_seq_name_#1_seq }
  }

\keys_define:nn { scontents / Scontents }
  {
    show-cmd  .code:n = { \keys_set:nn { scontents } { show-cmd = #1 } },
    save-cmd  .code:n = { \keys_set:nn { scontents } { save-cmd = #1 } }
  }

\ProvideDocumentCommand{ \Scontents }{ O{} +m }
  {
    \group_begin:
    \IfNoValueF {#1} { \keys_set:nn { scontents / Scontents } {#1} }
    \_scontents_append_contents:nn { \l_scontents_name_seq_cmd_tl } { #2 } % add to seq
    \bool_if:NT \l_scontents_show_cmd_bool
      {
        \_scontents_getfrom_seq:nn { -1 }{ \l_scontents_name_seq_cmd_tl }
      }
    \group_end:
  }

\ProvideExpandableDocumentCommand{ \getstored }{ O{1} m }
  {
    \_scontents_getfrom_seq:nn {#1} {#2}
  }

\ProvideDocumentEnvironment{ scontents }{}
  {
    \char_set_catcode_active:N \^^M
    \scontents_start_environment:w
  }
  {
    \scontents_stop_environment:
    \scontents_atend_environment:
  }

\cs_new_protected:Npn \scontents_environment_keys:w [#1]
  {
    \keys_set:nn { scontents } {#1}
  }

% Star environment
\group_begin:
\char_set_catcode_active:N \^^M
\cs_new_protected:Npn \scontents_start_environment:w #1 ^^M
  {
    \tl_if_blank:nF {#1} { \scontents_environment_keys:w #1 }
    \group_begin: % open group for env
    \use:c { filecontentsdefmacro } { \l_scontents_macro_tmp_tl } ^^M 
  }
\group_end:

% Stop environment
\cs_new_protected:Nn \scontents_stop_environment:
  {
    \endfilecontentsdefmacro 
    \group_end:  % close group for env
  }

% A variant to replace \^^M for \^^J (need by Verb{..} from fvextra)
\cs_generate_variant:Nn \tl_replace_all:Nnn { Nxx } 

% Expand \l_tmpa_tl and pass to seq
\cs_gset_protected:Nn \_scontents_macro_to_seq:
  { 
    \regex_replace_all:nnN { \^^M } { \^^J } \l_scontents_macro_tmp_tl
    \cs_log:N \l_scontents_macro_tmp_tl
    \exp_args:NNx \_scontents_append_contents:nn \l_scontents_name_seq_tl 
      {
       \exp_not:N \scantokens \exp_after:wN { \tl_use:c { l_scontents_macro_tmp_tl} } %
      }
  }

\cs_new_protected:Nn \scontents_atend_environment:
  {
    \_scontents_macro_to_seq:
    \bool_if:NT \l_scontents_show_env_bool
      {
        \_scontents_getfrom_seq:nn { -1 }{ \l_scontents_name_seq_tl }
      }
    \cs_undefine:N \l_scontents_macro_tmp_tl
  }
\ExplSyntaxOff
\begin{document}
\section{Test \Verb{\begin{scontents}[key=val]}}
Test \verb+\begin{scontents}+ no \verb+[key=val]+\par

\begin{scontents}
Using \Verb{scontents} env no \verb+[key=val]+, save in seq \verb+contents+ with index 1.\par
\begin{verbatim}
      (A) verbatim environment
\end{verbatim}
\end{scontents}

Test \verb+\begin{scontents}[show-env=true]+\par

\begin{scontents}[show-env=true]
Using \verb+scontents+ env with \Verb{[show-env=true]}, save in seq \verb+contents+ with index 2.
We have coded this in \LaTeX: $E=mc^2$.\par
\begin{Verbatim*}
      (B) verbatim environment
\end{Verbatim*}
\end{scontents}

Test \verb+\begin{scontents}[show-env=false]+\par

\begin{scontents}[show-env=false]
Using \verb+scontents+ env with \verb+[show-env=false]+, save in seq \verb+contents+ with index 3. 
We have coded this in \LaTeX: $E=mc^2$.\par
\begin{verbatim}
      (C) verbatim environment
\end{verbatim}
\end{scontents}

Test \verb+\begin{scontents}[show-env=true]+\par

\begin{scontents}[show-env=true]
Using \verb+scontents+ env with \verb+[show-env=true]+, save in seq \verb+contents+ with index 4.
We have coded this in \LaTeX: $E=mc^2$.\par
\begin{verbatim*}
      (D) verbatim environment
\end{verbatim*}
\end{scontents}

Test \verb+\Scontents{...}+\par
\Scontents{Using \texttt{Scontents} command (no verbatim), save in seq \texttt{contents} with index 5}

\section{Test \Verb{\getstored[index]{contents}}}
The total contents stored  in \verb+contents+ seq are \countseqsc{contents}.\par

\getstored[5]{contents}\par
\getstored{contents}

\section{A simple aplication using \Verb{pgffor} loop}
\newcounter{exeNr}
\newenvironment{exercise}
  {\refstepcounter{exeNr}\par\noindent This is exercise~\theexeNr}
  {\par}
\subsection{Exercises}
\begin{exercise}
\end{exercise}
\begin{scontents}[save-env = myansewer]
This is the answer to exercise 1, the shebang line for a Perl script

\begin{verbatim}
#!/usr/bin/env perl
\end{verbatim}
\end{scontents}

\begin{exercise}
\end{exercise}
\begin{scontents}[save-env = myansewer]
This is the answer to exercise 2
\end{scontents}

\begin{exercise}
\end{exercise}
\Scontents[save-cmd = myansewer]{This is the answer to exercise 3}

\subsection{Answers}
\newcounter{ansNr}
\newenvironment{answer}
  {\refstepcounter{ansNr}\par\noindent Answer~\theansNr:}
  {\par}

\foreach \i in {1,...,\countseqsc{myansewer}} {
\begin{answer}
\getstored[\i]{myansewer}
\end{answer}
}

\end{document}

An image of what the output is like:

Annexed query, expl3 provides two modules l3seq and l3prop to store data, is there any difference in this specific case?

Regards

Answer 1

Well, to start your code seems to add stuff to an expl3 seq variable (before the edit, now they are prop variables, but the difference is not much in terms of memory consumption). A seq variable is nothing but a macro which contains the items you added to that sequence. For example, after you do \seq_set_from_clist:Nn \l_tmpa_seq {a,b,c}, the seq variable will be a macro with:

\s__seq
\__seq_item:n {a}
\__seq_item:n {b}
\__seq_item:n {c}

An arbitrarily large sequence variable will be a macro with arbitrarily many \__seq_item:n {<stuff>} in it.

A prop list is similar. After you do \prop_set_from_keyval:Nn \l_tmpa_prop { a = x, b = y, c = zzz } the prop variable will be a macro with:

\s__prop
\__prop_pair:wn a\s__prop {x}
\__prop_pair:wn b\s__prop {y}
\__prop_pair:wn c\s__prop {zzz}

so an arbitrarily large prop list will have arbitrarily many \__prop_pair:wn <name>\s__prop {<value>} in it. the basic difference is that here you have to store not only a value, but also its name, so the overhead is a tad higher. While with the seq variable you had <stuff> plus three tokens (\__seq_item:n, {, and }), with a prop variable you have <name>, <value>, plus 4 more tokens (\__prop_pair:wn, \s__prop, {, and }).

Of course that once the contents of either start to grow, the overhead becomes insignificant, so the choice is up to the usage. If you want to store a sequence of things, then use a seq variable. If you want to store things and give a name to each use a prop list. Either will have a small memory overhead compared to their contents.

Now returning to your question, there is no imposed limit on the number of items a seq variable can contain because there is no imposed limit on the size of a \def in TeX. It can store arbitrarily large definitions as long as they fit in the imposed amount of memory available for TeX. Thus, the answer to your question will be similar to the answer of “how big a \def can TeX store in its memory?”.

---

If I simply load expl3 in a plain TeX file, create an (initially) empty seq variable, and ask for the memory stats:

\input expl3-generic.tex
\ExplSyntaxOn
\seq_new:N \l_test_seq
\ExplSyntaxOff
\tracingstats1
\bye

I get this (with pdfTeX in a TeXLive 2019):

Here is how much of TeX's memory you used:
 9137 strings out of 494647
 176410 string characters out of 6148463
 160755 words of memory out of 5000000
 10799 multiletter control sequences out of 15000+600000
 543492 words of font info for 60 fonts, out of 8000000 for 9000
 1416 hyphenation exceptions out of 8191
 27i,0n,59p,187b,317s stack positions out of 5000i,500n,10000p,200000b,80000s

XeTeX will show basically the same info, perhaps with a slight difference on the numbers, because it's fairly similar to pdfTeX. LuaTeX, on the other hand, will show a slightly different summary:

Here is how much of LuaTeX's memory you used:
 8789 strings out of 496318
 100000,154059 words of node,token memory allocated
 244 words of node memory still in use:
   3 hlist, 1 rule, 1 glyph, 27 glue_spec nodes
   avail lists: 2:10,3:3,5:3,7:2
 10892 multiletter control sequences out of 65536+600000
 60 fonts using 6755909 bytes
 61i,0n,59p,271b,446s stack positions out of 5000i,500n,10000p,200000b,100000s

because LuaTeX has a rather different memory management than its predecessors (see section 3.4.1 Memory allocation of the LuaTeX manual). However, the underlying information is fairly similar. The next paragraphs will be based on the output of pdfTeX, but the same principles apply to the other engines, changing the numbers a little bit. I'll also use a seq variable because it's easier to build up programatically than a prop variable, but the principles are all the same.

If I add an item to that seq I don't see any difference at all in the memory usage. In fact, if I add a bunch of items to that seq variable I don't see any difference up to a certain point. I think (not sure, though) that the memory stats showed by TeX are the maximum amount used during the run, but at some point it clears some memory and when I populate the seq it uses that memory which will not show in the stats.

However, once I pass that point anything that is added to the seq variable counts in the memory. This code barely passes that threshold:

\input expl3-generic.tex
\ExplSyntaxOn
\seq_new:N \l_test_seq
% \int_step_inline:nn {751} { \seq_put_right:Nn \l_test_seq { 0123456 } }% No difference
  \int_step_inline:nn {752} { \seq_put_right:Nn \l_test_seq { 0123456 } }
\ExplSyntaxOff
\tracingstats1
\bye

the memory usage is the same except for the words of memory, which increased from 160755 to 160769. The other memory types are left untouched by this operation. If I add another item to that variable (by changing the 752 to 753 the words of memory increase to 160789. Add another one and it goes to 160809, always in steps of 20. Why 20 you ask? Each added item is \__seq_item:n {0123456}, which are exactly 10 tokens (\__seq_item:n, {, 7 characters, and }), so this tells us that each token in a definition counts as two words of memory (perhaps because TeX represents a token by its character and category codes, so 2 words per token; but I'm just guessing here, sorry). If you were to use a prop list you would have to take into account the length of the name of each item, plus the extra tokens showed in the beginning of this answer.

Now we're getting somewhere. If I take the total words of memory available, subtract the number used, and divide by 20 I get an estimate of the number of 7-token items that I can still add to the sequence above, which is about 241962, which will take an awful lot of time to run :)

To reduce the time it takes to build that sequence you can enlarge each item. Here is a code that adds 997-token items to the seq variable, which added to the 3 tokens of the seq variable, grows the sequence in steps of 2000 words of memory:

\input expl3-generic.tex
\ExplSyntaxOn
\seq_new:N \l_test_seq
\int_step_inline:nn {2426} % Add 1 to see TeX blow up :)
  {
    \seq_put_right:Nn \l_test_seq
      { % Indentation is important even here :)
        0123456   012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
        0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
        0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
        0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
        0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
        0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
        0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
        0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
        0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
        0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
      }
  }
\ExplSyntaxOff
\tracingstats1
\bye

The code above brings the memory to 4999713 words of memory out of 5000000 (99.994% :).

---

Conclusion (?)

This all means that you can (roughly) store 2400 1000-token items in a seq variable when you are using pdfTeX or XeTeX. If you consider all the stored tokens to be ASCII characters, each 1000-token item is 1 kB, then you have approximately 2.4 MB of memory available for this seq variable (or any other 2.4 MB definition). Of course, the seq variable has an intrinsic memory footprint of 3 tokens, so the smaller the items stored, the less usable space you have.

If you are using LuaTeX, the memory usage will be roughly similar to the other engines. The test above with LuaTeX used 100000,5906145 words of node,token memory allocated (that is, 100000 words of node and 5906145 words of token memory, the latter is the one we're interested in), which is about 20% more than pdfTeX. However LuaTeX has dynamic memory allocation so, quoting the manual:

‘Out of main memory’ errors can still occur, but the limiting factor is now the amount of RAM in your system, not a predefined limit.

so yes, you can go pretty wild on the loop counter if you are using LuaTeX :)

For other engines the memory can be enlarged by changing the value of main_memory in texmf.cnf (default is 5000000 words, 2.5 MB).

However, should you reach this point in a document you are probably better off saving that to an external file. One situation where you can exceed the memory limit is with an environment that grabs its contents in a macro combined with an absurdly large content, like in “TeX capacity exceeded when externalizing”. However even in such cases something can be changed to drastically reduce memory usage somehow.