Symbol not found error when using Lua C .so module with LuaLaTeX

Question

I am trying to get a basic Lua C .so module working with LuaLaTeX.

I am able to run the library using my local Lua interpreter just fine (lua test.lua). But when I try to load it inside LuaLaTeX, I get the error as shown below.

The test file is test.tex. The Lua package is located in harfbuzz.sty, the lua file is luaharfbuzz.lua, and the C library code is in harfbuzz.c

My environment: Mac OS X, TeXLive 2015 and Lua 5.2.4.

The minimal code example is here: https://github.com/deepakjois/tex-sandbox/tree/master/luatex-lua-c-bridge

In the code above, I am compiling a file harfbuzz.c to harfbuzz.so. When I load it in my Lua code lua test.lua, it works fine. However, when I am loading it using LuaTeX using lualatex test.tex it throws an error.

What am I doing wrong?

Error Details:

lualatex test.tex
This is LuaTeX, Version beta-0.80.0 (TeX Live 2015) (rev 5238)
 restricted \write18 enabled.
(./test.tex
LaTeX2e <2015/01/01>
…
<snip>
…
(./luaharfbuzz.lua)
! LuaTeX error error loading module 'harfbuzz' from file './harfbuzz.so':
        dlopen(./harfbuzz.so, 6): Symbol not found: _luaL_setfuncs
  Referenced from: ./harfbuzz.so
  Expected in: flat namespace
 in ./harfbuzz.so
stack traceback:
        [C]: in ?
        [C]: in ?
        [C]: in function 'require'
        ./luaharfbuzz.lua:2: in main chunk
        [C]: in function 'require'
        [\directlua]:1: in main chunk.
\luacode@dbg@exec ...code@maybe@printdbg {#1} #1 }

l.6 \end{luacode*}

Update 1

Here is a comparision of the shared libraries that the lua and luatex binaries on my computer link to. Can that help explain the problem?

> otool -L `which lua` `which luatex`
/usr/local/bin/lua:
        /usr/local/lib/liblua.5.2.dylib (compatibility version 5.2.0, current version 5.2.4)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1225.1.1)
        /usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0)
/Library/TeX/texbin/luatex:
        /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)

Update 2

It seems the problem is that the luatex binary that comes with TeXLive strips a lot of symbols.

nm `which luatex` | grep _luaL_setfu
<…nothing…>

But I built my own version of luatex from trunk with the --nostrip flag enabled and got this

nm build/texk/web2c/luatex | grep _luaL_setfu
00000001002c54d0 T _luaL_setfuncs

So does that mean, there is no way to load a dynamically linked .so file with the default luatex on TeXLive 2015, because the _luaL_setfuncs cannot be located?

Answer 1

The problem turns out to be in the way LuaTeX is compiled when it is distributed. All the symbols are stripped out of the binary, so dynamically linked .so libraries can’t locate the symbols they need.

See the discussion on LuaTeX mailing list: http://tug.org/pipermail/luatex/2015-November/005440.html

The alternative is to compile your own LuaTeX binary from SVN, building it with the --nostrip flag, and then overwriting the existing LuaTeX binary with the new binary. Be careful with this step. Make sure you are compiling the right version of the library (ideally the one bundled with the TeXLive version), and run fmtutil --byengine luatex after copying over the binary to update the formats.