# Latex3 programming and tex primitives: how to "execute after" instead of expandafter

Here is a bit lengthy question. I hope it is not too specific.

I'm trying to scan some tex documents and I want to produce text files directly. I'll achieve this by scanning the tex input token by token. Characters with category code 11 or 12 are written to a string. Macros are expanded to unexpandables before continuing.

However when unexpandable primitive is encountered my code fails (probably because it enters infinite loop where primitive expands to itself). What I want to do is to pass unexpandable primitives along with their arguments to tex engine and then continue scanning. I'm just not sure how to achieve this. One way could be to define for each primitive a separate macro that scoops up the primitive and its arguments, then executes the primitive and continues with scanning. However maybe there is simpler solution.

First example document works and demonstrates what I want to do.

Second example fails because of \def primitive in document body.

First test document testdoc.tex:

\documentclass{article}

\def\HELLO#1{HELLO #1}

\begin{document}
Lorem ipsum dolor sit amet, consectetur adipiscing elit. \if ab A IS B \else A IS NOT B \fi. Nullam et volutpat nulla. \if cc C IS C \else C IS NOT C \fi. Suspendisse ultrices tortor eu elit hendrerit tristique. \HELLO{WORLD}. Praesent ut viverra mauris.
\end{document}


Result of first test document testdoc.txt:

Lorem ipsum dolor sit amet, consectetur adipiscing elit. A IS NOT B . Nullam et volutpat nulla. C IS C . Suspendisse ultrices tortor eu elit hendrerit tristique. HELLO WORLD. Praesent ut viverra mauris.

Second test document testdoc-2.tex:

\documentclass{article}

\begin{document}
Suspendisse ultrices tortor eu elit hendrerit tristique. \def\HELLO#1{HELLO #1} \HELLO{WORLD}. Praesent ut viverra mauris.
\end{document}


\ProvidesPackage{madtohtml}

\RequirePackage{xparse}
\RequirePackage{etoolbox}

\ExplSyntaxOn

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Initialise html string and create methods for appending stuff %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Start and stop scanning of tex input %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\NewDocumentCommand{\MYENDSCAN}{}{ENDENDENDENDENDEND}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Scanner: scan tex input token by token %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\cs_new:Nn\madtohtml_scanner_scan_start_a:{SCAN\c_space_tl STARTS}% Just for seeing if it works
\cs_new:Nn\madtohtml_scanner_scan_stop_a:N{SCAN\c_space_tl STOPS}% Just for seeing if it works

\token_if_primitive:NTF{#1}{%
\token_if_expandable:NTF{#1}%
}{%
\token_if_macro:NTF{#1}{\madtohtml_scanner_domacro:n{#1}}{%\def*ed or similar, or active char
NONE}}}}}%
}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%
%% Write html-file %%%%%%
\ExplSyntaxOn
}
\ExplSyntaxOff
\AfterEndDocument{
}
%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%

\AtBeginDocument{\everypar{\def\par{\MYENDSCAN\@@par}\MYSTARTSCAN}}
\let\@@enddocument\enddocument
\def\enddocument{\MYENDSCAN\def\par{\@@par}\@@enddocument}

\endinput


You may guess from my code that my ultimate goal is to produce html. Note that I'm very well aware of LaTeXML and Tralics. This is more for learning latex3 programmin and seeing what it can achieve.

• That way [producing html from TeX] madness lies! I tried a few times to do something similar before settling on my current system which involves a very complicated document class. Rather than scanning tokens, it allows them to process but redefines things at a lower level. It does use LaTeX3, though. You can see the code at github.com/loopspace/latex-to-internet Aug 4, 2018 at 13:22
• @LoopSpace I think I actually tested your code earlier. Some of the Latex3 interfaces have changed after your last update so it didn't work. I tried to patch it but didn't succeed. I think I'll revisit your code later and see if I can get it working. I'm also planning to redefine most macros and primitives. Aug 4, 2018 at 14:15
• The github repository is probably a bit behind my working directory. I'll update it soon. Aug 4, 2018 at 16:11

I'd use LuaLaTeX and hook into the pre_linebreak_filter. At this point everything has already been expanded but no hyphenation points inserted. Then I travese the nodes in the current paragraph and write the UTF-8 representation of all glyphs to the file. Glue is just replaced by spaces. The output is written to a file called export.txt.

Right now it is not recursively descendant, i.e. nested hlists and vlists are not handled. I'll leave this as an exercise.

\documentclass{article}

\usepackage{luacode}
\begin{luacode*}

local local_par_id = node.id("local_par")
local glue_id = node.id("glue")
local glyph_id = node.id("glyph")

local export_file = io.open("export.txt", "w")
local first_par = true

while n do
if n.id == local_par_id and not first_par then
export_file:write("\n\n")
elseif n.id == glyph_id then
export_file:write(utf8.char(n.char))
elseif n.id == glue_id then
export_file:write(" ")
end
n = n.next
first_par = false
end
end

end,
"export")
function() io.close(export_file) end,
"export")

\end{luacode*}

\def\HELLO#1{HELLO #1}

\begin{document}
Lorem ipsum dolor sit amet, consectetur adipiscing elit. \if ab A IS B \else A IS NOT B \fi. Nullam et volutpat nulla. \if cc C IS C \else C IS NOT C \fi. Suspendisse ultrices tortor eu elit hendrerit tristique. \HELLO{WORLD}. Praesent ut viverra mauris.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. \if ab A IS B \else A IS NOT B \fi. Nullam et volutpat nulla. \if cc C IS C \else C IS NOT C \fi. Suspendisse ultrices tortor eu elit hendrerit tristique. \HELLO{WORLD}. Praesent ut viverra mauris.
\end{document}


Contents of export.txt:

Lorem ipsum dolor sit amet, consectetur adipiscing elit. A IS NOT B . Nullam et volutpat nulla.  C IS C . Suspendisse ultrices tortor eu elit hendrerit tristique. HELLO WORLD. Praesent ut viverra mauris.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. A IS NOT B . Nullam et volutpat nulla.  C IS C . Suspendisse ultrices tortor eu elit hendrerit tristique. HELLO WORLD. Praesent ut viverra mauris.