3

I have a TeX Live 2019 distribution installed on Windows 10 and want to run a preloaded LaTeX based on e-TeX (with these packages among others: Calculator, Calculus, TikZ, CircuiTikZ) under WebAssembly in a web browser.

For the job I found TikZJax, which works as follows (quoted from readme.md by Jim Fowler kisonecat/tikzjax):

How does this work?

Using https://github.com/kisonecat/web2js the Pascal source of tex is compiled to WebAssembly; the latex format is loaded (without all the hyphenation data), and

\documentclass[margin=0pt]{standalone}
\def\pgfsysdriver{pgfsys-ximera.def}
\usepackage{tikz}

is executed. Then core is dumped; the resulting core is compressed, and by reloading the dumped core in the browser, it is possible to very quickly get to a point where TikZ can be executed. By using an SVG driver for PGF along with https://github.com/kisonecat/dvi2html the DVI output is converted to an SVG.

All of this happens in the browser.

I did the following steps according to instructions from Web2JS:

  1. download a clean copy of the TeX WEB sources; output: tex.web
  2. produce the Pascal source by tangle -ing, but with this changed version: tangle -underline tex.web etex.ch thanks to ShreevatsaR's tip; output: tex.p tex.pool after renaming from etex to tex
  3. compile the tex.p" to get the WebAssembly binary; output: out.wasm
  4. produce plain.fmt and a corresponding memory dump with a JavaScript named initex.js; input: out.wasm, plain.tex; output: core.dump, plain.fmt, plain.log, texput.log
  5. compile sample.tex; input: core.dump; output: sample.dvi, sample.log

I can't figure out how to make etex.ch right containing all changes for an eTeX built in Pascal (and running in WebAssembly).

I am not able to compile tex.p (which is actually an etex.p) with web2js to get the WebAssembly binary out.wasm.

I learned that in etex.ch are several changes missing, e.g. memory management.

Here`s an error as follows from a compiling try:

c:\texlive\eTeX\web2js\node_modules\binaryen\index.js:7
if(t){v=__dirname+"/";var ba,ca;a.read=function(c,e){var g=w(c);
g||(ba||(ba=require("fs")),ca||(ca=require("path")),
c=ca.normalize(c),g=ba.readFileSync(c)); return e?g:g.toString()};
a.readBinary=function(c){c=a.read(c,!0);c.buffer||
(c=new Uint8Array(c));assert(c.buffer);return c};1<process.argv.length&&
(a.thisProgram=process.argv[1].replace(/\\/g,"/"));
a.arguments=process.argv.slice(2);
process.on("uncaughtException",function(c){if(!(c instanceof x))
throw c;});process.on("unhandledRejection",y);a.quit=
                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ^
Need 32906 of memory

For this error to get I changed the following constants in tex.web before tangle-ing:

  • max_strings 3000 -> 500000
  • string_vacancies 8000 -> 90000
  • pool_size 32000 -> 6250000
  • max_halfword 65535 -> 268435455
  • mem_max 30000 -> 268435455
  • buf_size 500 -> 200000
  • stack_size 200 -> 5000
  • mem_top 3000 -> 268435455

Without these changes I get this error:

! You have to increase POOLSIZE.


How do I make the etex.ch right?


Update 04.08.2019


Thanks to Marcel Krüger's etex.sys I can now create a plain e-TeX without any problems.

Annotations on this very valuable answer:
- WebAssembly memory page size: 64 KiB i.e. 65536 Bytes [1]
- WebAssembly memory implementation limit: 2GB (as of today) => 32767 pages [2]


1 Allow providing more initial memory than specified by the module #540
2 Can not set TOTAL_MEMORY greater than 2Gb or expand the memory to greater than 2Gb

  • 1
    This is still "cutting-edge technology" (Fowler wrote an article about it in the latest TUGboat), so expect some work to get it exactly right. BTW 4 Jul 1776 12:00 is not a typo, it's just a default that Knuth left in tex.web for system-dependent changes to replace with a function for getting current date and time. And David Carlisle's answer there was partially a joke answer; I don't think it's recommended to hack your way like that instead of just using a latex format file. Also the example on tikzjax.com is already using etex with latex so it should be possible. – ShreevatsaR Jul 12 at 15:54
  • BTW you may also want to consider texlive.js or xetex.js instead of this method. Please update here if you get any of these methods to work well; I'm curious :-) – ShreevatsaR Jul 12 at 16:03
  • 1
    BTW I meant the example at people.math.osu.edu/fowler.291/latex not tikzjax.com. Anyway, yes, for e-TeX you need tex.web with etex.ch (e.g. tangle tex.web etex.ch I think), or you can try pdftex.web or even xetex.web from the TeX Live sources. – ShreevatsaR Jul 12 at 16:12
  • @ShreevatsaR TeXLive.Js is too slow. I will check XeTeX.Js. – CarpeDiemKopi Jul 12 at 16:13
  • 1
    My impression is that most of TeXLive.js's apparent slowness comes from having to fetch files over the network; it will be faster if you have the files pre-fetched or fetch them locally. XeTeX.js should be similar as it's based on texlive.js – ShreevatsaR Jul 12 at 16:41
4

You're increase of the pool size lead to additional memory requirements. So you do not need any other changes to eTeX, you have to increase the provided memory. In your Javascript versions, the amount of memory is set in the "compiler". For your settings you would need 32906 pages of memory, but there is an impmentation limit at 32767 pages. Luckily you can avoid this problem by using smaller values.

So we need to change some of the constants form etex.web. This doesn't mean that your etex.ch is "wrong" and you need a "right" one. Actually the license of etex.ch would forbid such modifications(At least without changing the name). Instead you should write a system dependent etex.sys file which you can pass to tangle later.

So first get copies from tex.web and etex.ch, then run

tie -m etex.web tex.web etex.ch

to get etex.web. Now you need a changefile with you new constants, for example save the following as etex.sys:

eTeX compatible constants for web2js

@x
@<Constants...@>=
@!mem_max=30000; {greatest index in \TeX's internal |mem| array;
  must be strictly less than |max_halfword|;
  must be equal to |mem_top| in \.{INITEX}, otherwise |>=mem_top|}
@!mem_min=0; {smallest index in \TeX's internal |mem| array;
  must be |min_halfword| or more;
  must be equal to |mem_bot| in \.{INITEX}, otherwise |<=mem_bot|}
@!buf_size=500; {maximum number of characters simultaneously present in
  current lines of open files and in control sequences between
  \.{\\csname} and \.{\\endcsname}; must not exceed |max_halfword|}
@!error_line=72; {width of context lines on terminal error messages}
@!half_error_line=42; {width of first lines of contexts in terminal
  error messages; should be between 30 and |error_line-15|}
@!max_print_line=79; {width of longest text lines output; should be at least 60}
@!stack_size=200; {maximum number of simultaneous input sources}
@!max_in_open=6; {maximum number of input files and error insertions that
  can be going on simultaneously}
@!font_max=75; {maximum internal font number; must not exceed |max_quarterword|
  and must be at most |font_base+256|}
@!font_mem_size=20000; {number of words of |font_info| for all fonts}
@!param_size=60; {maximum number of simultaneous macro parameters}
@!nest_size=40; {maximum number of semantic levels simultaneously active}
@!max_strings=3000; {maximum number of strings; must not exceed |max_halfword|}
@!string_vacancies=8000; {the minimum number of characters that should be
  available for the user's control sequences and font names,
  after \TeX's own error messages are stored}
@!pool_size=32000; {maximum number of characters in strings, including all
  error messages and help texts, and the names of all fonts and
  control sequences; must exceed |string_vacancies| by the total
  length of \TeX's own strings, which is currently about 23000}
@!save_size=600; {space for saving values outside of current group; must be
  at most |max_halfword|}
@!trie_size=8000; {space for hyphenation patterns; should be larger for
  \.{INITEX} than it is in production versions of \TeX}
@!trie_op_size=500; {space for ``opcodes'' in the hyphenation patterns}
@!dvi_buf_size=800; {size of the output buffer; must be a multiple of 8}
@!file_name_size=40; {file names shouldn't be longer than this}
@!pool_name='TeXformats:TEX.POOL                     ';
  {string of length |file_name_size|; tells where the string pool appears}
@.TeXformats@>

@ Like the preceding parameters, the following quantities can be changed
at compile time to extend or reduce \TeX's capacity. But if they are changed,
it is necessary to rerun the initialization program \.{INITEX}
@.INITEX@>
to generate new tables for the production \TeX\ program.
One can't simply make helter-skelter changes to the following constants,
since certain rather complex initialization
numbers are computed from them. They are defined here using
\.{WEB} macros, instead of being put into \PASCAL's |const| list, in order to
emphasize this distinction.

@d mem_bot=0 {smallest index in the |mem| array dumped by \.{INITEX};
  must not be less than |mem_min|}
@d mem_top==30000 {largest index in the |mem| array dumped by \.{INITEX};
  must be substantially larger than |mem_bot|
  and not greater than |mem_max|}
@y
@<Constants...@>=
@!mem_max=200000; {greatest index in \TeX's internal |mem| array;
  must be strictly less than |max_halfword|;
  must be equal to |mem_top| in \.{INITEX}, otherwise |>=mem_top|}
@!mem_min=0; {smallest index in \TeX's internal |mem| array;
  must be |min_halfword| or more;
  must be equal to |mem_bot| in \.{INITEX}, otherwise |<=mem_bot|}
@!buf_size=5000; {maximum number of characters simultaneously present in
  current lines of open files and in control sequences between
  \.{\\csname} and \.{\\endcsname}; must not exceed |max_halfword|}
@!error_line=72; {width of context lines on terminal error messages}
@!half_error_line=42; {width of first lines of contexts in terminal
  error messages; should be between 30 and |error_line-15|}
@!max_print_line=79; {width of longest text lines output; should be at least 60}
@!stack_size=1000; {maximum number of simultaneous input sources}
@!max_in_open=6; {maximum number of input files and error insertions that
  can be going on simultaneously}
@!font_max=75; {maximum internal font number; must not exceed |max_quarterword|
  and must be at most |font_base+256|}
@!font_mem_size=20000; {number of words of |font_info| for all fonts}
@!param_size=60; {maximum number of simultaneous macro parameters}
@!nest_size=40; {maximum number of semantic levels simultaneously active}
@!max_strings=60000; {maximum number of strings; must not exceed |max_halfword|}
@!string_vacancies=300000; {the minimum number of characters that should be
  available for the user's control sequences and font names,
  after \TeX's own error messages are stored}
@!pool_size=350000; {maximum number of characters in strings, including all
  error messages and help texts, and the names of all fonts and
  control sequences; must exceed |string_vacancies| by the total
  length of \TeX's own strings, which is currently about 23000}
@!save_size=600; {space for saving values outside of current group; must be
  at most |max_halfword|}
@!trie_size=8000; {space for hyphenation patterns; should be larger for
  \.{INITEX} than it is in production versions of \TeX}
@!trie_op_size=500; {space for ``opcodes'' in the hyphenation patterns}
@!dvi_buf_size=800; {size of the output buffer; must be a multiple of 8}
@!file_name_size=40; {file names shouldn't be longer than this}
@!pool_name='TeXformats:TEX.POOL                     ';
  {string of length |file_name_size|; tells where the string pool appears}
@.TeXformats@>

@ Like the preceding parameters, the following quantities can be changed
at compile time to extend or reduce \TeX's capacity. But if they are changed,
it is necessary to rerun the initialization program \.{INITEX}
@.INITEX@>
to generate new tables for the production \TeX\ program.
One can't simply make helter-skelter changes to the following constants,
since certain rather complex initialization
numbers are computed from them. They are defined here using
\.{WEB} macros, instead of being put into \PASCAL's |const| list, in order to
emphasize this distinction.

@d mem_bot=0 {smallest index in the |mem| array dumped by \.{INITEX};
  must not be less than |mem_min|}
@d mem_top==200000 {largest index in the |mem| array dumped by \.{INITEX};
  must be substantially larger than |mem_bot|
  and not greater than |mem_max|}
@z

@x
@d min_quarterword=0 {smallest allowable value in a |quarterword|}
@d max_quarterword=255 {largest allowable value in a |quarterword|}
@d min_halfword==0 {smallest allowable value in a |halfword|}
@d max_halfword==65535 {largest allowable value in a |halfword|}
@y
@d min_quarterword=0 {smallest allowable value in a |quarterword|}
@d max_quarterword=255 {largest allowable value in a |quarterword|}
@d min_halfword==0 {smallest allowable value in a |halfword|}
@d max_halfword==16777215 {largest allowable value in a |halfword|}
@z

Now you can run tangle:

tangle -underline etex.web etex.sys

You get the files etex.p and etex.pool.

Of course web2js will still look for tex.pool, but you can just change

filename = "tex.pool";

into

filename = "etex.pool";

in both header.js and library.js.

Now let's try

node compile.js etex.p

Similar to your original experiment, we get

[...]

Need 41 of memory

Now 41 is significantly less than 32906, especially it is below 32767. So we can just allocate more memory. This needs to be done consistently in four files: In index.js, initex.js, tex.js and pascal/program.js, change

var pages = 20;

into

var pages = 50;

(Probably 41 would be enough, but 50 looks nicer)

Now we can try

node compile.js etex.p

again. This time it actually works! You could use node initex.js now to get plain-TeX format, but we actually want eTeX. So you can get yourself a version of etex.src, etexdefs.lib and language.def and change

library.setInput("\nplain \\dump\n\n"

in initex.js into

library.setInput("\n*etex \\dump\n\n"

Here, the asterisk * is important, it enables the "extended mode". Also change &plain into &etex in the same file to preload etex.

Then

node initex.js

generates a e-TeX format etex.fmt and a memory dump, which can be used with

node tex.js
  • 1
    This is great! I was doing something similar. However, note that the OP wants to generate a LaTeX format, which is quite another beast entirely... seems to be quite painful, what with the zillions of files it requires (I got as far as some missing babel-latex.cfg file) and total lack of documentation anywhere on how to generate format files in a sane way. – ShreevatsaR Jul 17 at 21:06
  • @CarpeDiemKopi Sorry, I was not able to get the LaTeX format working... it's even more of a sprawling mess than I had imagined. – ShreevatsaR Jul 18 at 4:33
  • @ShreevatsaR That's sad. I was really looking forward to see that answer. – Marcel Krüger Jul 18 at 11:41
  • 1
    For what it's worth I managed to get a (somewhat) working LaTeX format (see my answer) — the main difference, from the last time I tried this, is that I automated the annoying step of finding the files. :-) – ShreevatsaR Aug 4 at 23:38
2

I managed to get a LaTeX format working with web2js, though with some caveats.

Here's a working (for me) sequence of steps.

  1. Get web2js: either download the zip file and unzip, or run

    git clone https://github.com/kisonecat/web2js.git
    
  2. Get tex.web: download using your browser, or run:

    wget http://mirrors.ctan.org/systems/knuth/dist/tex/tex.web
    
  3. Get etex.ch: download using your browser, or run:

    wget -O etex.ch 'https://tug.org/svn/texlive/trunk/Build/source/texk/web2c/etexdir/etex.ch?revision=32727&view=co'
    
  4. Tie them together:

    tie -m mytex.web tex.web etex.ch
    
  5. Make the following modifications to the resulting file (or you can use the “proper” way involving etex.sys etc., as in the answer by Marcel Krüger):

    @!mem_max=30000; {greatest index in \TeX's  |   @!mem_max=400000; {greatest index in \TeX'
    @!stack_size=200; {maximum number of simul  |   @!stack_size=1000; {maximum number of simu
    @!max_in_open=6; {maximum number of input   |   @!max_in_open=15; {maximum number of input
    @!max_strings=3000; {maximum number of str  |   @!max_strings=60000; {maximum number of st
    @!string_vacancies=8000; {the minimum numb  |   @!string_vacancies=300000; {the minimum nu
    @!pool_size=32000; {maximum number of char  |   @!pool_size=350000; {maximum number of cha
    @!trie_size=8000; {space for hyphenation p  |   @!trie_size=600000; {space for hyphenation
    @!trie_op_size=500; {space for ``opcodes''  |   @!trie_op_size=10000; {space for ``opcodes
    @d mem_top==30000 {largest index in the |m  |   @d mem_top==400000 {largest index in the |
    @d hash_size=2100 {maximum number of contr  |   @d hash_size=15000 {maximum number of cont
    @d hyph_size=307 {another prime; the numbe  |   @d hyph_size=2003 {another prime; the numb
    for i:=0 to @'37 do xchr[i]:=' ';           |   for i:=0 to @'37 do xchr[i]:=chr(i);
    for i:=@'177 to @'377 do xchr[i]:=' ';      |   for i:=@'177 to @'377 do xchr[i]:=chr(i);
    @d max_quarterword=255 {largest allowable   |   @d max_quarterword=65535 {largest allowabl
    @d max_halfword==65535 {largest allowable   |   @d max_halfword==16777215 {largest allowab
    

    These were determined most empirically, by bumping up the ones I got errors about. The change in the xchr assignments is as per the discussion at another question.

  6. Correspondingly, edit the four files index.js, initex.js, pascal/program.js and tex.js to change var pages = 20; to var pages=290;. (Actually, while playing with this I created a file commonMemory.js containing

    module.exports = { commonPages: function() { return 290; } };
    

    and used var pages = require('./commonMemory').commonPages(); or ... But that was just convenient while determining this number 290, and you don't have to do that.)

  7. Edit library.js: inside function reset, change this block:

        files.push({
          filename: filename,
          position: 0,
          descriptor: fs.openSync(filename,'r'),
        });
    

    to

        let basename = filename.slice(filename.lastIndexOf('/') + 1);
        const {spawnSync} = require('child_process');
        let realFilename = spawnSync('kpsewhich', [filename]).stdout.toString().trim();
        if (realFilename == '') {
            // try again with basename
            realFilename = spawnSync('kpsewhich', [basename]).stdout.toString().trim();
            if (realFilename == '') {
                // Give up, just create empty file
                spawnSync('touch', [basename]);
                realFilename = basename;
                console.log(`For filename #${filename}# created empty #${basename}#`);
            } else {
                console.log(`Found filename #${filename}# via basename at #${realFilename}#`);
            }
        } else {
            console.log(`Found filename #${filename}# at #${realFilename}#`);
        }
    
        files.push({
          filename: filename,
          position: 0,
          descriptor: fs.openSync(realFilename,'r'),
        });
    

    — the idea is that as creating a LaTeX format file loads zillions of files, some of which aren't even distributed with TeX Live, we make the file-lookup hook into kpsewhich to find all those files, and just leave the file empty if not found. For what it's worth, these were the files that were not found and for which empty files were used: babel-latex.cfg, il2enc.dfu, omlenc.dfu, omxenc.dfu, uenc.dfu.

  8. Edit initex.js to dump the LaTeX format instead of plain (and again when doing the core dump):

    -library.setInput("\nplain \\dump\n\n",
    +library.setInput("\n*latex.ltx \\dump\n\n",
    

    and

    -library.setInput("\n&plain\n\n",
    +library.setInput("\n&latex\n\n",
    
  9. Replace the contents of sample.tex with a LaTeX sample. For example, you can use (from here):

    \documentclass{article}
    \title{Cartesian closed categories and the price of eggs}
    \author{Jane Doe}
    \date{September 1994}
    \begin{document}
       \maketitle
       Hello world!
    \end{document}
    
  10. Get web2js dependencies and build its Pascal parser:

    npm install
    npm run-script build
    
  11. Build everything: from WEB (via TANGLE) to Pascal (via web2js) to WASM to loading and dumping format file and memory dump and then running TeX:

    tangle -underline mytex.web && \
    mv -f mytex.pool tex.pool && \
    node compile.js mytex.p && \
    node initex.js && \
    node tex.js
    

Note that sample.dvi has been created successfully and looks ok. So we have a working LaTeX format. You can try editing sample.tex and re-running node tex.js to typeset various LaTeX documents (to DVI).

Caveats:

  • Because of those missing files that were substituted, it's possible that hyphenation patterns for non-English languages, or those particular font encodings, may not work correctly. But I could not find these files even in the TeX Live sources so I'm not sure what they're supposed to contain, or whether they're expected to be empty anyway.

  • The first revision of this answer has a way to build a LaTeX format without increasing max_quarterword / max_halfword, or increasing the number of memory pages granted on the JS side. That came at the cost of not loading most languages' hyphenation patterns, and also is not sufficient for loading heavy-weight packages like TikZ. The current revision does not have those issues.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.