2

Here is a bit lengthy question. I hope it is not too specific.

I'm trying to scan some tex documents and I want to produce text files directly. I'll achieve this by scanning the tex input token by token. Characters with category code 11 or 12 are written to a string. Macros are expanded to unexpandables before continuing.

However when unexpandable primitive is encountered my code fails (probably because it enters infinite loop where primitive expands to itself). What I want to do is to pass unexpandable primitives along with their arguments to tex engine and then continue scanning. I'm just not sure how to achieve this. One way could be to define for each primitive a separate macro that scoops up the primitive and its arguments, then executes the primitive and continues with scanning. However maybe there is simpler solution.

First example document works and demonstrates what I want to do.

Second example fails because of \def primitive in document body.

First test document testdoc.tex:

\documentclass{article}

\usepackage{madtohtml}

\def\HELLO#1{HELLO #1}

\begin{document}
Lorem ipsum dolor sit amet, consectetur adipiscing elit. \if ab A IS B \else A IS NOT B \fi. Nullam et volutpat nulla. \if cc C IS C \else C IS NOT C \fi. Suspendisse ultrices tortor eu elit hendrerit tristique. \HELLO{WORLD}. Praesent ut viverra mauris.
\end{document}

Result of first test document testdoc.txt:

Lorem ipsum dolor sit amet, consectetur adipiscing elit. A IS NOT B . Nullam et volutpat nulla. C IS C . Suspendisse ultrices tortor eu elit hendrerit tristique. HELLO WORLD. Praesent ut viverra mauris.

Second test document testdoc-2.tex:

\documentclass{article}

\usepackage{madtohtml}

\begin{document}
Suspendisse ultrices tortor eu elit hendrerit tristique. \def\HELLO#1{HELLO #1} \HELLO{WORLD}. Praesent ut viverra mauris.
\end{document}

Excerpt from style file madtohtml.sty:

\ProvidesPackage{madtohtml}

\RequirePackage{xparse}
\RequirePackage{etoolbox}

\ExplSyntaxOn




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Initialise html string and create methods for appending stuff %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\str_new:N\g_madtohtml_htmlstring_str
\cs_new:Npn\madtohtml_htmlstring_append:n #1 {\str_gput_right:Nn\g_madtohtml_htmlstring_str{#1}}
\cs_generate_variant:Nn\madtohtml_htmlstring_append:n{x}
\NewDocumentCommand{\@madtohtml@htmlstring@append@n}{m}{\madtohtml_htmlstring_append:n{#1}}
\NewDocumentCommand{\@madtohtml@htmlstring@append@x}{m}{\madtohtml_htmlstring_append:x{#1}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Start and stop scanning of tex input %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\NewDocumentCommand{\MYSTARTSCAN}{}{\madtohtml_scanner_scan_start:}
\NewDocumentCommand{\MYENDSCAN}{}{ENDENDENDENDENDEND}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Scanner: scan tex input token by token %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\cs_new:Nn\madtohtml_scanner_scan_start:{\madtohtml_scanner_scan_start_a:\madtohtml_scanner_main:}
\cs_new:Nn\madtohtml_scanner_scan_start_a:{SCAN\c_space_tl STARTS}% Just for seeing if it works
\cs_new:Nn\madtohtml_scanner_scan_stop:{\madtohtml_scanner_scan_stop_a:N}
\cs_new:Nn\madtohtml_scanner_scan_stop_a:N{SCAN\c_space_tl STOPS}% Just for seeing if it works

\cs_new:Nn\madtohtml_scanner_main:{\peek_after:Nw\madtohtml_scanner_main_a:}%check what is next token
\cs_new:Nn\madtohtml_scanner_main_a:{\madtohtml_scanner_tests:n{\l_peek_token}}
\cs_new:Nn\madtohtml_scanner_tests:n{%
    \token_if_eq_meaning:NNTF{#1}{\MYENDSCAN}{\madtohtml_scanner_scan_stop:}{%
        \token_if_primitive:NTF{#1}{%
            \token_if_expandable:NTF{#1}%
                {\madtohtml_scanner_doexpprim:n{#1}}%
                {\madtohtml_scanner_dounexpprim:n{#1}}%
        }{%
            \token_if_macro:NTF{#1}{\madtohtml_scanner_domacro:n{#1}}{%\def*ed or similar, or active char
            \token_if_eq_catcode:NNTF{#1}{\c_catcode_other_token}{\madtohtml_scanner_dochar:n{#1}}{%
            \token_if_eq_catcode:NNTF{#1}{\c_catcode_letter_token}{\madtohtml_scanner_dochar:n{#1}}{%
            \token_if_space:NTF{#1}{\madtohtml_scanner_dospace:n{#1}}{%
        NONE}}}}}%
    }
}

\cs_new:Nn\madtohtml_scanner_doexpprim:n{\madtohtml_scanner_domacro:n{#1}}
\cs_new:Nn\madtohtml_scanner_dounexpprim:n{\madtohtml_scanner_domacro:n{#1}}
\cs_new:Nn\madtohtml_scanner_domacro:n{\madtohtml_scanner_domacro_a:N}
\cs_new:Nn\madtohtml_scanner_domacro_a:N{\exp_after:wN\madtohtml_scanner_main:#1}
\cs_new:Nn\madtohtml_scanner_dochar:n{\madtohtml_scanner_dochar_a:N}
\cs_new:Nn\madtohtml_scanner_dochar_a:N{#1\madtohtml_scanner_dochar_b:n{#1}\madtohtml_scanner_main:}
\cs_new:Nn\madtohtml_scanner_dochar_b:n{\madtohtml_htmlstring_append:n{#1}}
\cs_new:Nn\madtohtml_scanner_dospace:n{\madtohtml_scanner_dospace_a:N}
\cs_new:Nn\madtohtml_scanner_dospace_a:N{\c_space_tl\madtohtml_scanner_dospace_b:n{\c_space_tl}\madtohtml_scanner_main:#1}
\cs_new:Nn\madtohtml_scanner_dospace_b:n{\madtohtml_htmlstring_append:x{#1}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%
%% Write html-file %%%%%%
\ExplSyntaxOn
\iow_new:N\madtohtml_writeout_html_iow
\NewDocumentCommand{\@madtohtml@writeout@writehtml}{}{%
    \iow_open:Nn\madtohtml_writeout_html_iow{\c_sys_jobname_str .txt}
    \iow_now:Nx\madtohtml_writeout_html_iow{\str_use:N\g_madtohtml_htmlstring_str}
    \iow_close:N\madtohtml_writeout_html_iow
}
\ExplSyntaxOff
\AfterEndDocument{
    \@madtohtml@writeout@writehtml
}
%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%


\AtBeginDocument{\everypar{\def\par{\MYENDSCAN\@@par}\MYSTARTSCAN}}
\let\@@enddocument\enddocument
\def\enddocument{\MYENDSCAN\def\par{\@@par}\@@enddocument}


\endinput

You may guess from my code that my ultimate goal is to produce html. Note that I'm very well aware of LaTeXML and Tralics. This is more for learning latex3 programmin and seeing what it can achieve.

  • That way [producing html from TeX] madness lies! I tried a few times to do something similar before settling on my current system which involves a very complicated document class. Rather than scanning tokens, it allows them to process but redefines things at a lower level. It does use LaTeX3, though. You can see the code at github.com/loopspace/latex-to-internet – Loop Space Aug 4 '18 at 13:22
  • @LoopSpace I think I actually tested your code earlier. Some of the Latex3 interfaces have changed after your last update so it didn't work. I tried to patch it but didn't succeed. I think I'll revisit your code later and see if I can get it working. I'm also planning to redefine most macros and primitives. – Jarno_C-137 Aug 4 '18 at 14:15
  • The github repository is probably a bit behind my working directory. I'll update it soon. – Loop Space Aug 4 '18 at 16:11
3

I'd use LuaLaTeX and hook into the pre_linebreak_filter. At this point everything has already been expanded but no hyphenation points inserted. Then I travese the nodes in the current paragraph and write the UTF-8 representation of all glyphs to the file. Glue is just replaced by spaces. The output is written to a file called export.txt.

Right now it is not recursively descendant, i.e. nested hlists and vlists are not handled. I'll leave this as an exercise.

\documentclass{article}

\usepackage{luacode}
\begin{luacode*}
local local_par_id = node.id("local_par")
local glue_id = node.id("glue")
local glyph_id = node.id("glyph")

local export_file = io.open("export.txt", "w")
local first_par = true

local function export(head)
    local n = head
    while n do
        if n.id == local_par_id and not first_par then
            export_file:write("\n\n")
        elseif n.id == glyph_id then
            export_file:write(utf8.char(n.char))
        elseif n.id == glue_id then
            export_file:write(" ")
        end
        n = n.next
        first_par = false
    end
end

luatexbase.add_to_callback("pre_linebreak_filter",
                           function(head)
                               export(head)
                               return head
                           end,
                           "export")
luatexbase.add_to_callback("finish_pdffile",
                           function() io.close(export_file) end,
                           "export")
\end{luacode*}

\def\HELLO#1{HELLO #1}

\begin{document}
Lorem ipsum dolor sit amet, consectetur adipiscing elit. \if ab A IS B \else A IS NOT B \fi. Nullam et volutpat nulla. \if cc C IS C \else C IS NOT C \fi. Suspendisse ultrices tortor eu elit hendrerit tristique. \HELLO{WORLD}. Praesent ut viverra mauris.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. \if ab A IS B \else A IS NOT B \fi. Nullam et volutpat nulla. \if cc C IS C \else C IS NOT C \fi. Suspendisse ultrices tortor eu elit hendrerit tristique. \HELLO{WORLD}. Praesent ut viverra mauris.
\end{document}

Contents of export.txt:

Lorem ipsum dolor sit amet, consectetur adipiscing elit. A IS NOT B . Nullam et volutpat nulla.  C IS C . Suspendisse ultrices tortor eu elit hendrerit tristique. HELLO WORLD. Praesent ut viverra mauris. 

Lorem ipsum dolor sit amet, consectetur adipiscing elit. A IS NOT B . Nullam et volutpat nulla.  C IS C . Suspendisse ultrices tortor eu elit hendrerit tristique. HELLO WORLD. Praesent ut viverra mauris. 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.