5

We are writing a cls file that produces a text file as a side effect of running latex. This will be done in a cloud service within a docker container, so security is a serious concern. The output file will be processed by python, and it would be best to have the file encoded in UTF-8. This imposes imposes severe restrictions on pdflatex, which appears to only support the very limited output encodings of T1 etc (does anyone know how to read a file in python if it is encoded as T1?). If possible, I would like to avoid a dependency on luatex because of security issues (not a topic for this question, but it would probably delay us by six months). I have made quite a bit of progress in this, but the rest may require the latex3 APIs, which have relatively sparse documentation. I can find relatively few examples of code to process token lists or strings in the latex3 apis.

The root of the problem is that \write is designed to write out tokens, but I want only text. I will be discarding tokens where necessary in order to accomplish this goal. If you try \write{a~b} you end up with a token representation: a\protect \unhbox \voidb@x \protect \penalty \@M \ {}b This is unacceptable to us - we only want "a b" (the typesetting codes are meaningless outside of TeX). I would be willing to accept output that has all characters encoded as TeX strings like {"u}, since we could easily replace these with UTF-8 equivalents in python. Inline mathematics inside () or $$ should be preserved as text. Basically what I want is a macro like \newcommand{\mywrite}[1]{...} that does the following:

Input Output in file
\mywrite{This is~text} This is text
\mywrite{This has $a^2$ options} This has $a^2$ options
\mywrite{This has UTF-8 in Damgård} This has UTF-8 in Damgård
or: This has UTF-8 in Damg{\aa}rd
\mywrite{This has UTF-8 in Damg{\aa}rd (same as above)
\mywrite{This has {\bf bold}} This has bold
\mywrite{This has \textsl{slanted text} This has slanted text
\mywrite{This uses \amacro{text}} This uses <expanded from \amacro>
\mywrite{Uses a word\footnote{something} Uses a word
\mywrite{Uses a \smile} Uses a \smile
or: Uses a ⌣
\mywrite{Uses {ACM} for publishing} Uses ACM for publishing
\mywrite{Uses ``quotes''} Uses "quotes"
\mywrite{Wrapped lines treated Wrapped lines treated as spaces
as spaces}

If someone does \newcommand{\mybold}[1]{bold: {\bf #1}}, then \mywrite{This has \mybold{a}} should produce "this has bold: a". The rule of thumb is that any character listed in [Pakin's list](https://math.uoregon.edu/wp-content/uploads/2014/12/compsymb-1qyb3zd.pdfshould be encoded either as UTF-8 or as the corresponding TeX macro in the output. Any macro should be expanded to the point where such characters are recognized. Any inline mathematics should come through as a string as it would be written in TeX, bounded by () or $$. Display mathematics may also survive (but I'm much less concerned about this). Anything involving typesetting commands like font face, spacing, \twocolumn, \includegraphics, etc should be summarily dropped. This is intended to be used for things like author names, titles, section headings, and other metadata, but is NOT intended as a way to preserve typesetting commands. If there is doubt about a macro inside an argument, we drop it.

It seems like this should be possible to accomplish by scanning tokens in the parameter passed to \mywrite, each of which has a catcode 0-15. catcodes with character equivalents like 3, 7, 8, 10, 11, 12, 13 should have their corresponding character representations. The code should be designed to drop anything that does not have an obvious string equivalent.

4
  • (this is my first question in which I attempted to use a table. That part previewed ok, but does not show up in the question. Should we refine it?
    – mccurley
    Aug 29, 2022 at 22:36
  • 1
    you can write files in any encoding specifically look at hyperref writing unicode bookmark files or for newer code the l3 text mode gives standard commands for discarding tokens to get pure text and writing it as utf8 or utf16 or other encodings Aug 30, 2022 at 6:59
  • @mccurley the preview uses a different markdown rendering engine than the actual post, which creates many discrepancies in the rendering. Here the issue was that the horizontal line was added directly after the table, I inserted an extra newline and now the table renders ok. Side remark: {\bf bold} is deprecated, use \textbf{bold} instead.
    – Marijn
    Aug 30, 2022 at 9:07
  • Regarding the question itself: this seems complicated, because you want some macros expanded, others ignored but keeping contents, yet others completely dropped. Maybe you can save yourself some trouble by storing the relevant LaTeX code unchanged in the output file, add a skeleton document around it, and then convert to txt or some easily parseable format using Pandoc.
    – Marijn
    Aug 30, 2022 at 9:24

1 Answer 1

11

I think the standard LaTeX command \text_purify:n gets you most of what you need:

\documentclass{article}

\newwrite\txt
\immediate\openout\txt=\jobname.txt
\ExplSyntaxOn
\cs_new:Npn\mywrite#1{
\immediate\write\txt{\text_purify:n{#1}}
}
\ExplSyntaxOff
\begin{document}


\mywrite{hello world}
\mywrite{Γειά σου Κόσμε}
\mywrite{\textbf{this} \textit{that}}
\mywrite{$\alpha+\sqrt{x\beta^2}$}


\end{document}

Produces a .txt file with UTF-8 content:

hello world
Γειά σου Κόσμε
this that
$\alpha +\sqrt {x\beta ^2}$
4
  • Thank you for your suggestion David. I am constructing a cls file for a journal, and it has to be prepared for a lot of possible inputs. Whenever I use pdflatex, there seems to be no way to force UTF-8 output. Also, even if I have \usepackage[utf8]{inputenc}, if you put Γειά in extra.tex and \input{extra} then it contains about "LaTeX Error: Unicode character ε (U+03B5) not set up for use with LaTeX.". lualatex is satisfied with it. pdflatex seems to have character encoding too intertwined with fonts.
    – mccurley
    Aug 31, 2022 at 23:07
  • that inputenc message is unconnected to fonts and unconnected to file writing. I wrote that message ;) you are doing something wrong but as you have shown no code I can not guess what. @mccurley Sep 1, 2022 at 5:50
  • \begin{filecontents}{greek.tex} Γειά σου Κόσμε \end{filecontents} \documentclass{article} \usepackage[utf8]{inputenc} \begin{document} Character handling in pdflatex is a mystery. This works in lualatex. \input{greek.tex} \end{document}
    – mccurley
    Sep 2, 2022 at 7:10
  • @mccurley it only works in luatex if you use a font that has Greek. And in pdftex the part you are asking about in this question works fine. Γειά σου Κόσμε is written by pdftex in correctly encoded UTF-8 to greek.tex . The error you show is completely unrelated to writing files so unrelated to this question, you get the same from \documentclass{article}\begin{document}Γειά σου Κόσμε \end{document} and this has no error \documentclass{article}\usepackage[greek]{babel}\begin{document}Γειά σου Κόσμε \end{document} Sep 2, 2022 at 7:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .