Need macro to \write content into a file as a string from LaTeX

Question

We are writing a cls file that produces a text file as a side effect of running latex. This will be done in a cloud service within a docker container, so security is a serious concern. The output file will be processed by python, and it would be best to have the file encoded in UTF-8. This imposes imposes severe restrictions on pdflatex, which appears to only support the very limited output encodings of T1 etc (does anyone know how to read a file in python if it is encoded as T1?). If possible, I would like to avoid a dependency on luatex because of security issues (not a topic for this question, but it would probably delay us by six months). I have made quite a bit of progress in this, but the rest may require the latex3 APIs, which have relatively sparse documentation. I can find relatively few examples of code to process token lists or strings in the latex3 apis.

The root of the problem is that \write is designed to write out tokens, but I want only text. I will be discarding tokens where necessary in order to accomplish this goal. If you try \write{a~b} you end up with a token representation: a\protect \unhbox \voidb@x \protect \penalty \@M \ {}b This is unacceptable to us - we only want "a b" (the typesetting codes are meaningless outside of TeX). I would be willing to accept output that has all characters encoded as TeX strings like {"u}, since we could easily replace these with UTF-8 equivalents in python. Inline mathematics inside () or $$ should be preserved as text. Basically what I want is a macro like \newcommand{\mywrite}[1]{...} that does the following:

Input	Output in file
\mywrite{This is~text}	This is text
\mywrite{This has $a^2$ options}	This has $a^2$ options
\mywrite{This has UTF-8 in Damgård}	This has UTF-8 in Damgård
	or: This has UTF-8 in Damg{\aa}rd
\mywrite{This has UTF-8 in Damg{\aa}rd	(same as above)
\mywrite{This has {\bf bold}}	This has bold
\mywrite{This has \textsl{slanted text}	This has slanted text
\mywrite{This uses \amacro{text}}	This uses <expanded from \amacro>
\mywrite{Uses a word\footnote{something}	Uses a word
\mywrite{Uses a \smile}	Uses a \smile
	or: Uses a ⌣
\mywrite{Uses {ACM} for publishing}	Uses ACM for publishing
\mywrite{Uses ``quotes''}	Uses "quotes"
\mywrite{Wrapped lines treated	Wrapped lines treated as spaces
as spaces}

---

If someone does \newcommand{\mybold}[1]{bold: {\bf #1}}, then \mywrite{This has \mybold{a}} should produce "this has bold: a". The rule of thumb is that any character listed in [Pakin's list](https://math.uoregon.edu/wp-content/uploads/2014/12/compsymb-1qyb3zd.pdfshould be encoded either as UTF-8 or as the corresponding TeX macro in the output. Any macro should be expanded to the point where such characters are recognized. Any inline mathematics should come through as a string as it would be written in TeX, bounded by () or $$. Display mathematics may also survive (but I'm much less concerned about this). Anything involving typesetting commands like font face, spacing, \twocolumn, \includegraphics, etc should be summarily dropped. This is intended to be used for things like author names, titles, section headings, and other metadata, but is NOT intended as a way to preserve typesetting commands. If there is doubt about a macro inside an argument, we drop it.

It seems like this should be possible to accomplish by scanning tokens in the parameter passed to \mywrite, each of which has a catcode 0-15. catcodes with character equivalents like 3, 7, 8, 10, 11, 12, 13 should have their corresponding character representations. The code should be designed to drop anything that does not have an obvious string equivalent.

Answer 1

I think the standard LaTeX command \text_purify:n gets you most of what you need:

\documentclass{article}

\newwrite\txt
\immediate\openout\txt=\jobname.txt
\ExplSyntaxOn
\cs_new:Npn\mywrite#1{
\immediate\write\txt{\text_purify:n{#1}}
}
\ExplSyntaxOff
\begin{document}


\mywrite{hello world}
\mywrite{Γειά σου Κόσμε}
\mywrite{\textbf{this} \textit{that}}
\mywrite{$\alpha+\sqrt{x\beta^2}$}


\end{document}

Produces a .txt file with UTF-8 content:

hello world
Γειά σου Κόσμε
this that
$\alpha +\sqrt {x\beta ^2}$