3

In the spirit of questions like this, I'm trying to build a C version of TeX-the-program; specifically Taco's CXTeX (part of MetaTeX).

On Ubuntu 14 I can get it to compile, which results in a binary executable called cpdfetex. If you run it (./cpedetex), you get the welcome message

This is pdfeTeX, Version 3.141592-1.11b-2.1 (Web2C 7.5.2, C port 0.5.1)

However, if you try to do anything past that (like compile a .tex file as ./cpdfetex foo.tex, or type \end into the terminal), you get the error message

I can't find the format file `cpdfetex.efm'!

Any idea what to do?

(What's an .efm file supposed to contain, in the first place?)


Edit. The motivation is three-fold. I want a TeX distribution that is 1) barebones (just Knuth's plain TeX + Metafont), 2) fast (for a small file, I get 4ms compile times with KerTeX, 20ms with TeX Live's TeX, 30ms with eTeX, 40ms with pdfTeX, 120ms with LuaTex on dvi mode), and 3) hackable (preferably C-based, but at this point I'll take anything that compiles and runs...).

In particular, I want to access the final raw stream of bytes containing the .dvi output (right before it's written to disk) and pass it to a C program I'm writing (which understands the .dvi format). I'm currently writing the .dvi output to disk (using TeX-the-program) and reading it from disk (with the other program), but it'd be nicer to keep it all in memory.

  • 1
    Yes you need to generate/dump format files. I ran into this too, I've forgotten the exact details. – ShreevatsaR Aug 9 '17 at 2:22
  • 1
    Wild guess: can you try mktexfmt cpdfetex.efm and see if it works? (I really don't know why a typical TeX distribution is made so hard to build…) (BTW I recently wrote an answer that referred to the same question as you refer to; in particular the posts of Graham Douglas may help: readytext.co.uk/?cat=8 (or other categories on his blog).) – ShreevatsaR Aug 9 '17 at 3:24
  • 1
    How were you able to get it to compile BTW? I'm trying it out now (Ubuntu 17.04, gcc 6.3.3) and it fails with a compilation error. – ShreevatsaR Aug 9 '17 at 3:46
  • 1
    Yes I was able to compile it as well, with the same changes (2) and (3) except that instead of (1) using gcc-3.3 I renamed the variable cs_count in cpdfetex/pdftex/writet1.c. And now I'm stuck in the same way as you :-) BTW, Graham Douglas mentions he did build and get CxTeX working at some point, in these two posts. So it's definitely possible, I just don't understand enough. But trying to build LuaTeX may be an easier alternative today. – ShreevatsaR Aug 9 '17 at 6:56
  • 1
    Not an answer to this question, but just FYI: Your question reminded me: I just asked (and answered) a question about another TeX implementation, NTS written in Java, here. I don't know your specific motivation for trying CXTeX, but some of it may be addressed by either NTS or LuaTeX (or one of the other TeX implementations that actually exist!) – ShreevatsaR Aug 10 '17 at 1:04
5

Firstly: LuaTeX is a descendant of CXTeX. Why do I say this? I cannot read Dutch, but I notice certain similarities between these two papers both by Taco Hoekwater: De CXTeX distributie (MAPS 30, Voorjar (Spring) 2004) and LuaTEX says goodbye to Pascal (MAPS 39, EuroTeX 2009). :-) So when the latter paper says:

In the winter of 2008–2009, we invested a lot of time in hand-converting the entire LuaTeX code base into a set of C source files that are much closer to current programming practices. The big WEB file has been split into about five dozen pairs of C source files and include headers.

I imagine that this effort was influenced / helped by the author's earlier experience with CXTeX. So you may want to just use LuaTeX (which too is written in C). It has had more development and testing, and is known to work.


Here's how I was able to get CXTeX to work, somewhat (at least for non-math), using the “data” files from a TeX Live distribution. (Ubuntu 17.04, gcc 6.3.3.)

  1. Create a directory for cxtex, get sources, unpack them:

    mkdir cxtex && cd cxtex
    wget http://metatex.org/cxtex/cxtex-source-0.51.tar.gz
    tar xvfz cxtex-source-0.51.tar.gz
    cd cxtex-0.51/
    
  2. In cpdfetex/pdftex/writet1.c, globally rename cs_count to some other name (I used cs_count_here). Reason: There's a conflict between

    EXTERN int cs_count; /* total number of known identifiers */
    

    on line 29 of cpdfetex/hash.h, and cpdfetex/pdftex/writet1.c which has

    static int cs_count, cs_size, cs_size_pos;
    

    on line 238, besides other mentions of cs_count.

  3. In texutil/perlemu.h and texutil/perlemu.c and texutil/texexec.c, globally rename strndup to some other name consistently (I used strndup_here). Reason: there's a standard function called strndup in libc. (Note: You may also be able to just get rid of the definition in strndup in the files texutil/perlemu.{h,c}, so that that texutil/texexec.c uses the standard libc function. It will probably work, but I haven't tried this.)

  4. Edit cpdfetex/types.h to change

    typedef long int integer;
    

    to

    typedef int integer;
    

    Reason: When dumping format files, the function dump_int uses type integer, while undump_int seems to be often called with something of type int. When I was compiling it, for my combination of compiler and machine architecture, we had integer being 8 bytes and int being 4 bytes, which is incompatible. There may be other and better solutions to this problem, but this is what I did.

  5. In Makefile (the top-level one, inside cxtex/cxtex-0.51), remove the mentions of the .exe files: change

    all:
        cd texk/kpathsea && $(MAKE)
        cd cpdfetex && $(MAKE)         && cp cpdfetex.exe cpdfetex ../built
        cd texutil  && $(MAKE) texutil && cp texutil.exe  texutil  ../built
        cd texutil  && $(MAKE) texexec && cp texexec.exe  texexec  ../built
    

    to

    all:
        cd texk/kpathsea && $(MAKE)
        cd cpdfetex && $(MAKE)         && cp cpdfetex ../built
        cd texutil  && $(MAKE) texutil && cp texutil  ../built
        cd texutil  && $(MAKE) texexec && cp texexec  ../built
    

    Reason: We aren't building on Windows, so there won't be any .exe files.

With these changes, we are ready to compile:

make

Note that there are lots of warnings, including warnings of undefined behaviour and array out-of-bounds accesses. This indicates the codebase isn't quite perfect. When make is done, three binaries are created in built/: cpdfetex, texexec, and texutil. There's still some work before they can be used.

  1. Move to a particular directory: I just went up one level, to my cxtex directory:

    cd ..
    
  2. Create a pdftex.cfg file:

    touch pdftex.cfg
    
  3. Copy plain.tex:

    cp /usr/share/texlive/texmf-dist/tex/plain/base/plain.tex .
    
  4. Copy all the files it references (fonts and hyphenation):

    cp /usr/share/texlive/texmf-dist/fonts/tfm/public/cm/{cmr{10,9,8,7,6,5},cmmi{10,9,8,7,6,5},cmsy{10,9,8,7,6,5},cmex10,cmss10,cmssq8,cmssi10,cmssqi8,cmbx{10,9,8,7,6,5},cmtt{10,9,8},cmsltt10,cmsl{10,9,8},cmti{10,9,8,7},cmu10,cmmib10,cmbsy10,cmcsc10,cmssbx10,cmdunh10}.tfm .
    cp /usr/share/texlive/texmf-dist/fonts/tfm/public/knuth-lib/manfnt.tfm .
    cp /usr/share/texlive/texmf-dist/tex/generic/hyphen/hyphen.tex .
    
  5. Dump format file:

    ./cxtex-0.51/built/cpdfetex -ini plain.tex '\dump'
    

This creates a file called plain.efm. And now cxtex is usable!

Create a test file:

    echo "hello \bye" > hello.tex

and run it through TeX!

    ./cxtex-0.51/built/cpdfetex -efm=plain hello.tex

I've tried this with a few example plain-TeX files like "story.tex" and even "xii.tex", it works. But when I tried gentle.tex, it only got through four pages (one of them happens to be an empty page) before failing with:

! This can't happen (mlist4).
\Big ...eft #1\vbox to11.5\p@ {}\right .\n@space $
                                                  }}
l.477 \line{4.~$\Bigl\{
                       $Groups, $\bigl\{$Groups, $\{$and More%

So there are some bugs in the code (possibly ones we introduced with our changes), that still need to be debugged. Happy hacking!

The cool thing is that (unlike NTS) even LaTeX can work:

  1. Copy these files from the TeX Live sources: texmf-dist/tex/latex/base/* and texmf-dist/fonts/tfm/public/latex-fonts/*.tfm.

  2. Run

    ./cxtex-0.51/built/cpdfetex -ini
    

    and at the ** prompt type * and Enter, then

    \input latex.ltx
    

    This dumps the format to file texput.efm (couldn't figure out how to change that), which if you which you can rename to latex.efm.

Then with a file test.tex like:

\documentclass{article}
\begin{document}

This is a document.

Hello world.

This is math $x$ and $y$.

This is also math:

$$z$$

\end{document}

you can run:

./cxtex-0.51/built/cpdfetex -efm=texput test.tex

(or efm=latex if you renamed it).

The moment there is anything nontrivial in math mode (anything more than a single letter or digit), it seems to be buggy and either crash or give weird errors. Now that the source code to your TeX program is readable C, you can have fun debugging those. :-)

  • What an answer! Thanks so much! Sadly, when I try to run ./cxtex-0.51/built/cpdfetex -efm=plain hello.tex, I get the error: (Fatal format file error(); I'm stymied). But this approach looks promising, since I finally know explicitly how to generate an .efm file, and what it should contain. – étale-cohomology Aug 11 '17 at 12:55
  • 1
    @étale-cohomology I was getting exactly that error before I made the typedef int integer change. Note that you can pass -debug-format which will tell a little bit more. – ShreevatsaR Aug 11 '17 at 13:22
  • Yes, that fixed that error! The command cxtex-0.51/built/cpdfetex -ini plain.tex '\dump' generates No pages of output, but it does output a plain.efm file (size: 227k). (However, pdftex.cfg is still 0k in size, which makes we wonder what purpose it serves!) Now, when I run cxtex-0.51/built/cpdfetex -efm=plain hello.tex, I get the error: This is pdfeTeX, Version 3.141592-1.11b-2.1 (Web2C 7.5.2, C port 0.5.1 (./hello.tex{pdftex.cfg}Segmentation fault (core dumped) – étale-cohomology Aug 11 '17 at 14:03
  • 1
    Sorry to hear about the segmentation fault; guess the two versions of the compiler generated different code for the undefined behaviour. I guess you can work through it in a debugger… let me know if you fix it! – ShreevatsaR Aug 11 '17 at 16:53
  • 1
    @étale-cohomology As I understand it, it's a config file that the program expects: it fails if the file isn't there, but it's ok for the file to be empty. Searching through the code, it appears to be read in cxtex-0.51/cpdfetex/pdftex/config.c function readconfigfile, and can contain options like page_height, output_format, pdf_minorversion and the like. – ShreevatsaR Aug 11 '17 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.