1

Problem Description

As in the title.

MWE:

\documentclass{article}
\begin{document}

% first write some file content for demo...
\begin{filecontents*}{test test.txt}
line 1  !?#\xyz
line 2
\end{filecontents*}

% what to do here? I want to read the whole content of the file into variable \result

\ExplSyntaxOn
\str_show:N \result  % → the whole content of the file.
\ExplSyntaxOff

\end{document}
6
  • I found several very-similar questions (see links below), but none that is exactly identical to this one.
    – user202729
    Commented Jun 10, 2022 at 13:25
  • Side note, if the file is huge, you may want to refrain from using this method.
    – user202729
    Commented Jun 10, 2022 at 13:26
  • TeX doesn't work with strings, so "string variable" in nonsense from TeX point of view. There are only token lists in TeX.
    – wipet
    Commented Jun 10, 2022 at 17:27
  • 1
    @wipet The question title contains “(expl3) string variable”. String variables are perfectly well defined in expl3. The question didn't mention that computational complexity was a concern, but its “string variable” part, at least, is clear. In case it is not for you, see “The l3str package: Strings” in interface3.pdf.
    – frougon
    Commented Jun 10, 2022 at 19:03
  • @frougon Because expl3 is based on TeX and TeX doesn't work with strings, something like "string variable" is only mystification and denial of TeX principles. This can bring only misunderstanding. Moreover \string primitive is something different, so misunderstanding is very probable.
    – wipet
    Commented Jun 10, 2022 at 19:26

2 Answers 2

5

One option is to use \file_get:nnN. There are other options as well.

%! TEX program = pdflatex
\documentclass{article}
\begin{document}

\begin{filecontents*}{test test.txt}
line 1  !?#\xyz
line 2
\end{filecontents*}


\ExplSyntaxOn

\file_get:nnN {test~test.txt} {
    % alternatively (although the effects the different on LuaTeX)
    %   \int_step_inline:nnn {0} {255} {\char_set_catcode_other:n{#1}}
    \cctab_select:N \c_other_cctab
    \endlinechar=10~
    % ↑ this must be done after the cctab line
    %   because cctab changes the value of endlinechar as well
} \result
\str_set:NV \result \result

\str_show:N \result  % → the whole content of the file.
\ExplSyntaxOff

\end{document}

This saves the whole content of the file into variable \result, each "new line character" in the file will be represented with a character 10.

Explanation

\file_get:nnN {test~test.txt} {

Use the command, as explained. Note that since this is expl3 environment the space need to be written as ~.

    \cctab_select:N \c_other_cctab

Set the catcode. On XeTeX it's impractical to set for all the characters in 0..1114111, so there exists corner cases where one of the characters in the file has catcode 1/2 and make the content unbalanced thus cause an error.

Note that this command, in addition to setting the catcode table as its name suggests, it also sets the endlinechar (in this catcode table endlinechar = -1), as such we want char with charcode 10 to separate the lines we need to explicitly set it below.

    \endlinechar=10~

Set endlinechar to 10, so each new line character in the file will be represented with a character 10 to be precise, a token with char code 10 and catcode 12 in the resulting string.

This is expl3 environment it's good practice to specify the space explicitly to terminate the number (or use \scan_stop: / \relax but I don't like that name it's longer. There's also \int_set:Nn \endlinechar {10} but that... relies on implementation details...?)

} \result
\str_set:NV \result \result

Detokenize the result. This step is important, because in order to get the result as a string, tokens with charcode 32 (space) should have catcode 10 (space), while it would have catcode 12 (other) as above.

Limitations

  • First (for XeTeX/LuaTeX engine only), if there happens to be some character with char code ≥ 256 and some unusual catcode (e.g. catcode 1 -- open brace) it might break. Most of the time the detokenization can handle it however.
  • If the file to be read is a TeX file, synctex data on that file might be "lost". (the details is complex.)
  • Trailing spaces on each line are stripped (and maybe trailing tabs as well).

Related questions

4
  • Side note, the str_show might print something that doesn't make sense depends on your operating system, because character 10 is a CR or LF, something like that.
    – user202729
    Commented Jun 10, 2022 at 13:32
  • 1) In the paragraph starting with “Detokenize the result”, there is a mix between char code and catcode (swap needed). 2) \str_show:n should be \str_show:N. 3) I get “Undefined control sequence” for \pretty:V. I can fix these if you want...
    – frougon
    Commented Jun 26, 2022 at 21:04
  • @frougon Ah, that \pretty thing is an internal debugging macro for me, I keep forget to deleting them. Otherwise good points.
    – user202729
    Commented Jun 27, 2022 at 0:10
  • Would be nice to make a package out of this code
    – yegor256
    Commented Oct 7, 2022 at 5:34
1

I propose to use \ior_str_map_variable:NNn:

\begin{filecontents*}{test.txt}
line 1  !?#\xyz
line 2
\end{filecontents*}

\documentclass{article}

\ExplSyntaxOn
\ior_new:N \g__my_ior
\str_new:N \l__my_str

\cs_new_protected:Npn \my_verb_read_into:Nn #1#2
  {
    \str_clear_new:N #1   % define #1 if necessary, then clear it

    \ior_open:Nn \g__my_ior {#2}
    \ior_str_map_variable:NNn \g__my_ior \l__my_str
      {
        % Precaution in case the LINE FEED char ^^J might be active
        \str_put_right:Nx #1 { \l__my_str \char_generate:nn { 10 } { 12 } }
      }
    \ior_close:N \g__my_ior
  }

\NewDocumentCommand \verbReadInto { m m }
  {
    \my_verb_read_into:Nn #1 {#2}
  }
\ExplSyntaxOff

\begin{document}

\verbReadInto{\result}{test.txt}
\show\result

\end{document}

which shows:

\result=macro:
->line 1  !?#\xyz
line 2
.

All lines, including the last one, are terminated by a LINE FEED character (10) with catcode 12.

Of course, one could use \g_tmpa_ior and \l_tmpa_str in \my_verb_read_into:Nn (and spare the declarations of \g__my_ior and \l__my_str), but be careful not to shoot yourself in the foot when the code grows with new functions that use the same scratch variables.

Actually, if it were not for space reasons in this tight space for the code block, I'd have included most of the function name in the names of these two variables, for the same reason.

8
  • Although note that this method takes time quadratic in the number of lines in the file. There's also \tl_build_put* functions (they're still quadratic) and e.g. \tokapp primitives etc.
    – user202729
    Commented Jun 10, 2022 at 14:26
  • Because of the repeated \str_put_right:Nx? I would hope it is cheap enough, but otherwise you could append each line separately to a seq variable (I hope this is linear but haven't checked).
    – frougon
    Commented Jun 10, 2022 at 14:36
  • At the moment (my TeXlive version at least) seq is represented as \s__seq\__seq_item:n{1}\__seq_item:n{2}\__seq_item:n{3} so each change need to change the whole thing, quadratic as well.
    – user202729
    Commented Jun 10, 2022 at 14:36
  • Yes, I have now looked at the code of \seq_put_right:Nn and I believe \str_put_right:Nn has the same complexity (time and memory) as \seq_put_right:Nn. That said, whether it is quadratic depends on the complexity of expanding a macro in function of, say, the number of tokens inside the macro (judging from the code of \tl_put_right:Nn). I don't know how it is done in TeX impementations, just wondering if you are too pessimistic.
    – frougon
    Commented Jun 10, 2022 at 14:44
  • Just a short note: l3candidates has \tl_build_put_right:Nx and friends that might be relevant here (“\tl_build_begin:N Clears the ⟨tl var⟩ and sets it up to support other \tl_build_... functions, which allow accumulating large numbers of tokens piece by piece much more efficiently than standard l3tl functions”).
    – frougon
    Commented Jun 26, 2022 at 8:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .