How to run CXTeX?

Question

In the spirit of questions like this, I'm trying to build a C version of TeX-the-program; specifically Taco's CXTeX (part of MetaTeX).

On Ubuntu 14 I can get it to compile, which results in a binary executable called cpdfetex. If you run it (./cpedetex), you get the welcome message

This is pdfeTeX, Version 3.141592-1.11b-2.1 (Web2C 7.5.2, C port 0.5.1)

However, if you try to do anything past that (like compile a .tex file as ./cpdfetex foo.tex, or type \end into the terminal), you get the error message

I can't find the format file `cpdfetex.efm'!

Any idea what to do?

(What's an .efm file supposed to contain, in the first place?)

---

Edit. The motivation is three-fold. I want a TeX distribution that is 1) barebones (just Knuth's plain TeX + Metafont), 2) fast (for a small file, I get 4ms compile times with KerTeX, 20ms with TeX Live's TeX, 30ms with eTeX, 40ms with pdfTeX, 120ms with LuaTex on dvi mode), and 3) hackable (preferably C-based, but at this point I'll take anything that compiles and runs...).

In particular, I want to access the final raw stream of bytes containing the .dvi output (right before it's written to disk) and pass it to a C program I'm writing (which understands the .dvi format). I'm currently writing the .dvi output to disk (using TeX-the-program) and reading it from disk (with the other program), but it'd be nicer to keep it all in memory.

Answer 1

Firstly: LuaTeX is a descendant of CXTeX. Why do I say this? I cannot read Dutch, but I notice certain similarities between these two papers both by Taco Hoekwater: De CXTeX distributie (MAPS 30, Voorjar (Spring) 2004) and LuaTEX says goodbye to Pascal (MAPS 39, EuroTeX 2009). :-) So when the latter paper says:

In the winter of 2008–2009, we invested a lot of time in hand-converting the entire LuaTeX code base into a set of C source files that are much closer to current programming practices. The big WEB file has been split into about five dozen pairs of C source files and include headers.

I imagine that this effort was influenced / helped by the author's earlier experience with CXTeX. So you may want to just use LuaTeX (which too is written in C). It has had more development and testing, and is known to work.

---

Here's how I was able to get CXTeX to work, somewhat (at least for non-math), using the “data” files from a TeX Live distribution. (Ubuntu 17.04, gcc 6.3.3.)

1. Create a directory for cxtex, get sources, unpack them:

   mkdir cxtex && cd cxtex
   wget http://metatex.org/cxtex/cxtex-source-0.51.tar.gz
   tar xvfz cxtex-source-0.51.tar.gz
   cd cxtex-0.51/
   

2. In cpdfetex/pdftex/writet1.c, globally rename cs_count to some other name (I used cs_count_here). Reason: There's a conflict between

   EXTERN int cs_count; /* total number of known identifiers */
   

   on line 29 of cpdfetex/hash.h, and cpdfetex/pdftex/writet1.c which has

   static int cs_count, cs_size, cs_size_pos;
   

   on line 238, besides other mentions of cs_count.

3. In texutil/perlemu.h and texutil/perlemu.c and texutil/texexec.c, globally rename strndup to some other name consistently (I used strndup_here). Reason: there's a standard function called strndup in libc. (Note: You may also be able to just get rid of the definition in strndup in the files texutil/perlemu.{h,c}, so that that texutil/texexec.c uses the standard libc function. It will probably work, but I haven't tried this.)

4. Edit cpdfetex/types.h to change

   typedef long int integer;
   

   to

   typedef int integer;
   

   Reason: When dumping format files, the function dump_int uses type integer, while undump_int seems to be often called with something of type int. When I was compiling it, for my combination of compiler and machine architecture, we had integer being 8 bytes and int being 4 bytes, which is incompatible. There may be other and better solutions to this problem, but this is what I did.

5. In Makefile (the top-level one, inside cxtex/cxtex-0.51), remove the mentions of the .exe files: change

   all:
       cd texk/kpathsea && $(MAKE)
       cd cpdfetex && $(MAKE)         && cp cpdfetex.exe cpdfetex ../built
       cd texutil  && $(MAKE) texutil && cp texutil.exe  texutil  ../built
       cd texutil  && $(MAKE) texexec && cp texexec.exe  texexec  ../built
   

   to

   all:
       cd texk/kpathsea && $(MAKE)
       cd cpdfetex && $(MAKE)         && cp cpdfetex ../built
       cd texutil  && $(MAKE) texutil && cp texutil  ../built
       cd texutil  && $(MAKE) texexec && cp texexec  ../built
   

   Reason: We aren't building on Windows, so there won't be any .exe files.

With these changes, we are ready to compile:

make

Note that there are lots of warnings, including warnings of undefined behaviour and array out-of-bounds accesses. This indicates the codebase isn't quite perfect. When make is done, three binaries are created in built/: cpdfetex, texexec, and texutil. There's still some work before they can be used.

1. Move to a particular directory: I just went up one level, to my cxtex directory:

   cd ..
   

2. Create a pdftex.cfg file:

   touch pdftex.cfg
   

3. Copy plain.tex:

   cp /usr/share/texlive/texmf-dist/tex/plain/base/plain.tex .
   

4. Copy all the files it references (fonts and hyphenation):

   cp /usr/share/texlive/texmf-dist/fonts/tfm/public/cm/{cmr{10,9,8,7,6,5},cmmi{10,9,8,7,6,5},cmsy{10,9,8,7,6,5},cmex10,cmss10,cmssq8,cmssi10,cmssqi8,cmbx{10,9,8,7,6,5},cmtt{10,9,8},cmsltt10,cmsl{10,9,8},cmti{10,9,8,7},cmu10,cmmib10,cmbsy10,cmcsc10,cmssbx10,cmdunh10}.tfm .
   cp /usr/share/texlive/texmf-dist/fonts/tfm/public/knuth-lib/manfnt.tfm .
   cp /usr/share/texlive/texmf-dist/tex/generic/hyphen/hyphen.tex .
   

5. Dump format file:

   ./cxtex-0.51/built/cpdfetex -ini plain.tex '\dump'
   

This creates a file called plain.efm. And now cxtex is usable!

Create a test file:

    echo "hello \bye" > hello.tex

and run it through TeX!

    ./cxtex-0.51/built/cpdfetex -efm=plain hello.tex

I've tried this with a few example plain-TeX files like "story.tex" and even "xii.tex", it works. But when I tried gentle.tex, it only got through four pages (one of them happens to be an empty page) before failing with:

! This can't happen (mlist4).
\Big ...eft #1\vbox to11.5\p@ {}\right .\n@space $
                                                  }}
l.477 \line{4.~$\Bigl\{
                       $Groups, $\bigl\{$Groups, $\{$and More%

So there are some bugs in the code (possibly ones we introduced with our changes), that still need to be debugged. Happy hacking!

The cool thing is that (unlike NTS) even LaTeX can work:

1. Copy these files from the TeX Live sources: texmf-dist/tex/latex/base/* and texmf-dist/fonts/tfm/public/latex-fonts/*.tfm.

2. Run

   ./cxtex-0.51/built/cpdfetex -ini
   

   and at the ** prompt type * and Enter, then

   \input latex.ltx
   

   This dumps the format to file texput.efm (couldn't figure out how to change that), which if you which you can rename to latex.efm.

Then with a file test.tex like:

\documentclass{article}
\begin{document}

This is a document.

Hello world.

This is math $x$ and $y$.

This is also math:

$$z$$

\end{document}

you can run:

./cxtex-0.51/built/cpdfetex -efm=texput test.tex

(or efm=latex if you renamed it).

The moment there is anything nontrivial in math mode (anything more than a single letter or digit), it seems to be buggy and either crash or give weird errors. Now that the source code to your TeX program is readable C, you can have fun debugging those. :-)