How to build a LuaTeX binary with debug info?

Question

I would like to build a binary of LuaTeX with debug info, so that it can be run inside a debugger (like gdb) and one can step through the program in terms of the source code. My goal is to debug the LuaTeX executable purely for gaining knowledge of its internals.

I don't understand how formats stuff works, and it seems like it's getting in the way of setting up debugging environment.

Here is what I tried.

• Compiled luahbtex. Pretty easy, works right from the repo.
• Changed /usr/local/texlive/2021/bin/x86_64-linux/lualatex link to point to the compiled luahbtex.
• lualatex mwe.tex complains it cannot find texmf.cnf where luahbtex is located. I copied texmf.cnf and changed TEXMFROOT to /usr/local/texlive/2021.
• I ran fmtutil -user --all

Still all I can see is

$ lualatex mwe.tex
This is LuaHBTeX, Version 1.13.0 (TeX Live 2021)
 restricted system commands enabled.

(Fatal format file error; I'm stymied)

It works if I copy my luahbtex to /usr/local/texlive/2021/bin/x86_64-linux, but it's not very convenient to change the system wide executable.

Answer 1

Note: this answer concerns the special case that you only want to compile the one executable luahbtex and also preserve the original one, while not duplicating the other executable/helper scripts, instead of compiling/installing the whole TeXLive, which is what the asker of this particular question is trying to do (unsuccessfully).

Because this method are not officially supported by the scripts, you may expect encountering several hard-to-understand errors. If all fails, just do the official way and consume a few GBs of additional hard disk space/wait a few hours.

The official way is roughly

• Download the TeXLive source code
• Read the files named README.*, especially README.2building and README.3installing (as mentioned above, you need to download a lot of support files separately and must install it before using)

---

1. Build the executable luahbtex

It's possible to simply download TeXLive source code and run ./Build --debug inside.

Note: after building, check if the binary have debug symbols first. (for example with gdb <executable>). Last time I tried /inst/bin/x86_64-pc-linux-gnu/luahbtex does not have debug symbols, /Work/texk/web2c does.

Optionally set TL_MAKE_FLAGS environment variable. For example -j5.

Alternatively, to compile only LuaTeX: (there are two bugs in the current compilation commands that requires fixing.)

Adapting from section "4.6 Build one engine" in README.2building file, run:

set -e # bash: fail on error, just in case

# fix bug 1
mkdir Work

cd Work        
../configure --without-x --disable-shared --disable-all-pkgs \
    --enable-luahbtex --disable-synctex -C CFLAGS=-g CXXFLAGS=-g
make -j5

# fix bug 2 pplib manually
cd libs/pplib
make -j5
cd ../..

cd texk/web2c
make -j5 luahbtex

The compilation takes no more than 5 minutes on a 4-core machine, with the configuration part takes much of the time.

• Note: bug 1: mkdir Work is missing from README.2building. Need to be added manually.

• Note: bug 2: for some reason compiling only LuaTeX will not compile pplib. So you need to compile pplib manually first.

  Otherwise you get the error message

  ../../../texk/web2c/luatexdir/luamd5/md5lib.c:197:10: fatal error: utilsha.h: No such file or directory
    197 | #include "utilsha.h"
        |          ^~~~~~~~~~~
  compilation terminated.
  

  Also discussed in https://www.mail-archive.com/dev-luatex@ntg.nl/msg04722.html but no solution there.

To build luahbtex instead, change all occurrences of luatex to luahbtex. (and because of another bug, you need to run make -j5 luahbtex again after it finishes, because for some reason currently it builds luaharfbuzz before the harfbuzz library.) Otherwise you get the error message

In file included from ../../../texk/web2c/luatexdir/luaharfbuzz/src/luaharfbuzz/blob.c:1:
../../../texk/web2c/luatexdir/luaharfbuzz/src/luaharfbuzz/luaharfbuzz.h:4:10: fatal error: hb.h: No such file or directory
    4 | #include <hb.h>
      |          ^~~~~~
compilation terminated.

(For the difference, it's explained in man luatex:

The luahbtex variant can use the HarfBuzz engine for glyph shaping, instead of LuaTeX's built-in shaper.

)

2. Use the executable

2.1. Copy to global location

This one is the easiest method, but don't blindly copy paste code from the Internet.

sudo mv -i /usr/bin/luahbtex /usr/bin/luahbtex__backup
sudo cp ./luahbtex /usr/bin/luahbtex
mktexfmt lualatex

The executable uses the called file name to determine what to do, so luahbtex → lualatex makes a difference. (see Difference between luatex and lualatex binaries for more details)

2.2. Otherwise

Still requires global copy. (actually it may be possible to modify user-local fmtutil.cnf, but I don't want to figure this one out)

cp ./luahbtex /usr/bin/luahbtex__debug
cd /usr/bin
sudo ln -rs luahbtex__debug  lualatex__debug

then

vim $(kpsewhich fmtutil.cnf)

add the line

lualatex__debug luahbtex__debug language.dat,language.dat.lua lualatex.ini

Then it can be used as lualatex__debug. For some reason it recompiles the format file every time however.

Reason: by strace output, the executable looks for the fmt file in ~/.texlive/texmf-var/web2c/luahbtex/lualatex__debug.fmt while every time it's generated at ~/.texlive/texmf-var/web2c/luahbtex__debug/lualatex__debug.fmt. As a patch (again...) it's possible to do this...

cd ~/.texlive/texmf-var/web2c/luahbtex/
ln -rs ../luahbtex__debug/lualatex__debug.fmt .

2.3. Without global modification at all

This method is not officially supported so the information are mostly by trial and error, may break at any time. Also before editing anything, read the comment at the top of the file.

Because of how the scripts detect the location of the executables, it's quite complex/fragile to get things right. (it's not intended to be used this way anyway)

• While you have to use the new lualatex executable to generate the format file, you have to use the old mktexfmt, otherwise it will fail with

  Can't locate mktexlsr.pl in @INC (@INC contains: X/inst/tlpkg X/inst/texmf-dist/scripts/texlive /usr/lib/perl5/5.34/site_perl /usr/share/perl5/site_perl /usr/lib/perl5/5.34/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/5.34/core_perl /usr/share/perl5/core_perl) at X/mktexfmt line 23.
  BEGIN failed--compilation aborted at X/mktexfmt line 25.
  

  (because according to the explanation in https://github.com/conda-forge/texlive-core-feedstock/issues/19#issuecomment-384236207, it's necessary to install some additional things to make mktexfmt (alias fmtutil) work.)

• Without that, with a global TeX Live, it's also possible to use the global mktexfmt instead by making sure that one is being used.

  Unlike what the comment claims, it's not sufficient to set the $PATH.

  From the file $(kpsewhich -var-value=TEXMFROOT)/tlpkg/TeXLive/TLUtils.pm:

  sub prepend_own_path {
    my $bindir = dirname(Cwd::abs_path(which('kpsewhich')));
    if (win32()) {
      $bindir =~ s!\\!/!g;
      $ENV{'PATH'} = "$bindir;$ENV{PATH}";
    } else {
      $ENV{'PATH'} = "$bindir:$ENV{PATH}";
    }
  }
  

  So the actual location of kpsewhich (its original location if it's a symlink) matters.

  This one I have no idea how to fix. cp $(which kpsewhich) ., make a bash script named kpsewhich in the current folder that calls the real one, cp /usr/share/texmf-dist/web2c/texmf.cnf ., export PATH="$(realpath .):$PATH", mktexfmt lualatex all fails.