Struggling to Implement FileSystem Based Includes

Question

I'm trying to follow along with this answer to automagically include files based on the filesystem hierarchy.

My file structure looks like:

article.tex
sections/
|-- 00-introductions.tex
|-- 10-data.tex
|-- 20-analysis.tex
|-- 90-conclusions.tex
|-- data/
|   |-- 00-xxx.tex 
|   |-- NN-xxx.tex
|
|-- analysis/
    |-- 00-xxx.tex
    |-- NN-xxx.tex

So I want to include everything inside sections in article.tex, everything in sections/data in 10-data.tex, and everything in sections/analysis in 20-analysis.tex. For the purposes of this question, let's just focus on article.tex.

In an attempt to follow the linked answer, I've created the following bash script to generate a list of the appropriate files:

#! /bin/bash
ls -mp sections/ | grep -v / | tr ' ' '\n' > sections.list

And I've added this to my .latexmkrc so that it runs on every build (this bit works). It outputs the following (the new line is there if that's relevant):

00-introduction.tex,
10-data.tex,
20-analysis.tex,
30-conclusions.tex,

I've also tried an alternative that didn't have the commas appended.

My articles.tex (sans cruft) looks like this:

\documentclass[twocolumn, a4paper]{article}

\usepackage{pgffor}

\newcommand*{\listofsections}{\input{sections.list}}

\begin{document}

\foreach \c in \listofsections {%
    \input{sections/\c}%
}%

\end{document}

And if I just output \listofsections it looks fine to me:

It looks as though something's wrong in my definition of the loop as these errors are popping up in the logs:

/home/james/repos/discos-paper/article.tex:28: Missing endcsname inserted. let l.28 } % /home/james/repos/discos-paper/article.tex:28: Use of ??? doesn't match its definition. ???
! LaTeX3 Error: let invalid in file name. Lost: reserved@d =... l.28 } % /home/james/repos/discos-paper/article.tex:28: Missing endcsname inserted. let l.28 } % /home/james/repos/discos-paper/article.tex:28: Use of ??? doesn't match its definition. ???
! LaTeX3 Error: let invalid in file name. Lost: reserved@d =... l.28 } % /home/james/repos/discos-paper/article.tex:28: Missing endcsname inserted. let l.28 } % /home/james/repos/discos-paper/article.tex:28: Use of ??? doesn't match its definition. ???
! LaTeX3 Error: let invalid in file name. Lost: reserved@d =... l.28 } %

! LaTeX Error: File `sections/.tex' not found.

What have I missed here?

---

Edit 1:

For some reason, it seems as though my \listofsections is coming through as a single item and the iterator is only being called once...

I've also now tried editing the list file to be:

00-introduction.tex,10-data.tex,20-analysis.tex,30-conclusions.tex

Which still doesn't work, nor does it work if I place that in the \newcommand but curiously does if I copy the contents straight into the foreach loop... So I'm really not sure what's going wrong if I'm honest.

Answer 1

if sections.list has one filename per line (no comma)

00-introduction.tex
10-data.tex
20-analysis.tex
30-conclusions.tex

You can simply read one line at a time to get the filename, then input:

\documentclass{article}
\newread\sections
\openin\sections=sections.list
\begin{document}

\loop
{\endlinechar=-1 \global\read\sections to \next}
\ifx\next\empty\else
\input{\next}
\repeat

\end{document}

Answer 2

I don't think that \foreach is the right tool, as it executes every item in the loop in a group.

You aren't defining \listofsections as the list, but as a command that inputs a file.

\begin{filecontents*}{\jobname-sections.list}
00-introduction.tex,
10-data.tex,
20-analysis.tex,
30-conclusions.tex,
\end{filecontents*}

\documentclass[twocolumn, a4paper]{article}

\usepackage{pgffor}
\usepackage{catchfile}

\CatchFileDef{\listofsections}{\jobname-sections.list}{}

\begin{document}

\foreach \c in \listofsections {%
  \if\relax\c\relax
  \else
    %\input{sections/\c}%
    I would input ``sections/\c''\endgraf
  \fi
}

\end{document}

I changed the name just to keep filenames distinguished in my “tryout” folder and used a “mock \input'' just to show what happens. The \if\relax\c\relax is needed to avoid a spurious final input.

My preferred way would be

\begin{filecontents*}{\jobname-sections.list}
00-introduction.tex,
10-data.tex,
20-analysis.tex,
30-conclusions.tex,
\end{filecontents*}

\documentclass[twocolumn, a4paper]{article}

\ExplSyntaxOn

\NewDocumentCommand{\inputfromlist}{m}
 {% #1 is the name of the file containing the list
  % store the contents of the file in \l_tmpa_tl
  \file_get:nnN { #1 } { } \l_tmpa_tl
  % transform the tl into a clist
  \clist_set:NV \l_tmpa_clist \l_tmpa_tl
  % cycle through the items i the list
  \clist_map_inline:Nn \l_tmpa_clist
   {
    %\input{##1}
    I~would~input~``sections/##1''\par
   }
 }

\ExplSyntaxOff

\begin{document}

\inputfromlist{\jobname-sections.list}

\end{document}

The function \file_get:nnN is the expl3 analog of \CatchFileDef. I then transform the contents into a clist (empty items are automatically removed) and map it.

What does \clist_set:NV \l_tmpa_clist \l_tmpa_tl do? It is the same as

\clist_set:Nn \l_tmpa_clist { <contents of \l_tmpa_tl> }

The \clist_set:Nn function does a normalization, so empty items (that is, consisting of zero or more spaces) are removed.

Finally, the \clist_map_inline:Nn function loops over the list's items. The current item is denoted by #1, but, since we're inside the definition of a command, we must use ##1.

Answer 3

If doing this without packages I might have TeX switch to verbatim-catcode-régime with \endlinechar denoting the number of the code-point of the carriage-return-character and input the file list via expandable \@@input and process it as a list of carriage-return-delimited arguments.

(The internal character encoding scheme with traditional TeX is ASCII. The internal character encoding scheme with LuaTeX and XeTeX is unicode whereof ASCII is a strict subset. The number of the code-point of the carriage-return-character is 13 both in ASCII and in unicode. In .tex-input-files the carriage-return-character can be denoted in TeX's ^^-notation via ^^M; M is the 13^th letter in the latin alphabet. ^^-notation requires category-code 7(superscript) to be assigned to the character ^, which is the default.)

This way filenames are read under verbatim-catcode-régime. This might be a feature in case filenames contain sequences of space characters and/or characters like ^ etc that usually are/might be treated in special ways by TeX or—like { and }—need to be balanced when occurring within macro-arguments like the argument of the macro \input.

\futurelet is used only to ensure that TeX looks ahead and hereby removes the eof-token at the end of the file which otherwise might trigger a file ended while scanning the use of...-error when \inputfileloop grabs the last/appended \relax-dummy-line which marks the end of the loop. (Alternatively one could use \everyeof for appending the last dummy-line.)

\begin{filecontents*}{\jobname_inputfiles.txt}
00-introduction.tex
10-data.tex
20-analysis.tex
30-conclusions.tex
40-weird}%}  {^#^^61^^!.tex
\end{filecontents*}

\documentclass{article}

\begingroup
\makeatletter\endlinechar=-1\relax\catcode`\^^M=12 %
\@firstofone{%
  \endgroup
  \NewDocumentCommand\inputfiles{mv}{%
     \begingroup
     \let\do\@makeother\dospecials\do\^^M\do\^^I%
     %\endlinechar=`\^^M\relax %<- This is the default, thus this line is turned into a comment.
     \def\temp{\inputfileloop{}{#2}}%
     \expandafter\futurelet\expandafter\tempb\expandafter\temp\@@input "#1" \relax^^M%
  }%
  \@ifdefinable\inputfileloop{%
    % recursively in #1 accumulate a list of \input-commands
    %   #1 list of \input-commands gathered so far
    %   #2 directory
    %   #3 filename or \relax-token to process in this iteration.
    \long\def\inputfileloop#1#2#3^^M{%
      \ifx\relax#3\expandafter\@secondoftwo\else\expandafter\@firstoftwo\fi
      % Use this for inputting:
      %{\def\temp{\inputfileloop{#1\input{#2#3}}{#2}}\futurelet\tempb\temp}{\endgroup#1}%
      % Use this for message on terminal instead of inputting:
      {\def\temp{\inputfileloop{#1\message{I would \string\input{#2#3}^^J}}{#2}}\futurelet\tempb\temp}{\endgroup#1}%
    }%
  }%
}%

\begin{document}

\inputfiles{\jobname_inputfiles.txt}{section/}

\end{document}

Messages on terminal are:

I would \input{section/00-introduction.tex}
I would \input{section/10-data.tex}
I would \input{section/20-analysis.tex}
I would \input{section/30-conclusions.tex}
I would \input{section/40-weird}%}  {^#^^61^^!.tex}

With the last message the argument of the \input-command is:
section/section/40-weird}%}␣␣{^#^^61^^!.tex
(␣ denotes a space character.)

The token-sequence forming that argument is tokenized under verbatim-category-code-régime, so

• several spaces don't collapse into a single one.
• ^^-notation is disabled, i.e., ^^61^^! is not tokenized as aa
• { and } are not treated as braces for scoping/denoting arguments but are treaated as ordinary characters.
• # is treated as an ordinary character which can be used without doubling within arguments of macros which define scratch-macros from their arguments.
• % is treated as an ordinary character

---

It just occurred to me that you might like to crank out empty lines when the file holding the list of input-files is processed:

% Let's within the current directory/folder create the file holding
% the list of input-files:

\begin{filecontents*}{\jobname_inputfiles.txt}
00-introduction.tex
10-data.tex
20-analysis.tex
30-conclusions.tex

mysubsection/40-bla.tex
50-weird}%}  {^#^^61^^!.tex
\end{filecontents*}


\documentclass{article}

\begingroup
\makeatletter\endlinechar=-1\relax\catcode`\^^M=12 %
\@firstofone{%
  \endgroup
  \NewDocumentCommand\inputfiles{mv}{%
     \begingroup
     \let\do\@makeother\dospecials\do\^^M\do\^^I%
     %\endlinechar=`\^^M\relax %<- This is the default, thus this line is turned into a comment.
     \def\temp{\inputfileloop{}{#2}}%
     \expandafter\futurelet\expandafter\tempb\expandafter\temp\@@input "#1" \relax^^M%
  }%
  \@ifdefinable\inputfileloop{%
    % recursively in #1 accumulate a list of \input-commands
    %   #1 list of \input-commands gathered so far
    %   #2 directory;  slash etc needs to be included so you can leave it empty if you like.
    %   #3 filename or \relax-token to process in this iteration.
    \long\def\inputfileloop#1#2#3^^M{%
      \ifx\relax#3\expandafter\@secondoftwo\else\expandafter\@firstoftwo\fi
      {%
        \def\temp{%
          \ifcat$\detokenize{#3}$\expandafter\@gobble\else\expandafter\inputfileappendtolist\fi
          {#3}\inputfileloop{#1}{#2}%
        }%
        \futurelet\tempb\temp
      }{\endgroup#1}%
    }%
  }%
  \newcommand\inputfileappendtolist[4]{%
    #2{#3%
      % Use this for inputting:
      %\input{#4#1}%
      % Use this for message on terminal instead of inputting:
      \message{I would \string\input{#4#1}^^J}%
    }{#4}%
  }%
}

\begin{document}

\inputfiles{\jobname_inputfiles.txt}{mysection/}

\end{document}

Messages on terminal are:

I would \input{mysection/00-introduction.tex}
I would \input{mysection/10-data.tex}
I would \input{mysection/20-analysis.tex}
I would \input{mysection/30-conclusions.tex}
I would \input{mysection/mysubsection/40-bla.tex}
I would \input{mysection/50-weird}%}  {^#^^61^^!.tex}

With the last message the argument of the \input-command is:
mysection/50-weird}%}␣␣{^#^^61^^!.tex
(␣ denotes a space character.)