What is "the" Lua root for context packages?

Question

As package author, I want to write a Lua module which can be found by both lualatex and context.

The lualatex prototype works reasonably and I am convinced that this will also be true if I release the package.

However, the context prototype does not find my files, but I can force it to find my files by specifying a suitable LUAINPUTS environment variable (which points to the directory from which my files can be found directly).

However, I do not understand if (or how) my package will work in the "wild" where this environment variable is not set as in my development environment.

The question is essentially: where should I place my files such that context can require it?

Here is what I believe to be a minimal working example:

Let us assume that my package resides in /tmp/luamodule/texmf/tex/generic/pgfplots/libs/foo/bar/xyz.lua . This resembles my vague idea that texmf will be the global texmf entry point, then I have my package (directories tex/generic/pgfplots/libs) and then I have a sub-directory structure resembling my Lua package (foo/bar/xyz.lua).

The package file xyz.lua contains the advanced package

io.write("\nFOUND IT!\n")

Then I have a file containing

\documentclass{standalone}

\directlua{require('foo.bar.xyz')}

\begin{document}

OK.
\end{document}

Processing this with

export LUAINPUTS="/tmp/luamodule/texmf/tex/generic/pgfplots//:"
lualatex xyz.tex

works: I see "FOUND IT" in the console output.

Next, I retry the same with context and write a file xyzcontext.tex containing

\directlua{require('foo.bar.xyz')}
\starttext
OK.
\stoptext

Translating this with

export LUAINPUTS="/tmp/luamodule/texmf/tex/generic/pgfplots//:"
context xyzcontext

fails because it cannot find the Lua file.

However, it works if I write

export LUAINPUTS="/tmp/luamodule/texmf/tex/generic/pgfplots/libs/:"
context xyzcontext

The difference is that my first LUAINPUTS path contains a recursive include (//) whereas the second one contains the precise root directory.

I have learned already that context does things differently compared to lualatex (does not use kpse during require and handles periods differently). But this issue lets we wonder: lualatex appears to work with recursively defined lua file paths. Judging from the texmf.cnf configs shipped with tex live, I would expect that my directory naming scheme will work. But what with context!? Will it find my files if I do not set such a root path? It seems that this kind of naming scheme works will with PGF's graph drawing lib (in both lualatex and context), but I don't really see why.

I even tried to copy my .lua files into the TL 2014 global texmf tree (followed by texhash) -- without success.

Thus, my question: where should I place my .lua files such that both lualatex and context can find them?

related: Best Practices for Lua Modules

Answer 1

For the sake of having a community wiki answer that can be upvoted to take this question off, and that I might reformat if it gets enough upvotes/requests, here are the comments:

Did you run mtxrun --generate to recreate the filename database? That should make files in the TEXMF available to Context’s lookup routines. – Philipp Gesang Dec 25 '14 at 15:53

[… continued] Also, Context code rarely calls require() to load packages, usually only for external, non-TeX libs. Instead it’s more idiomatic to use environment.loadluafile() at the Lua end, and \registerctxluafile from TeX. – Philipp Gesang Dec 25 '14 at 15:53

@phg I was unware of mtxrun --generate. I ran it right now without any apparent difference (i.e. it still does not find the file). – Christian Feuersänger Dec 25 '14 at 17:31

@phg your reference to environment.loadluafile is new to me. Is that the solution? I have also received input from some context guy that dofile(resolvers.findfile("foo.bar.whatever.lua")) is (part of?) a best-practice. Admittedly, this only increases my confusion on how to write LUA libraries which are supposed to be used from within both lualatex and context. – Christian Feuersänger Dec 25 '14 at 17:34

... perhaps an answer to my question might be a switch of sorts "if I am context else if I am lualatex ..." – Christian Feuersänger Dec 25 '14 at 17:35

If the file isn’t found after --generate then perhaps it’s located in the wrong tree? Modules from the Garden usually reside in the texmf-modules, stuff that you install manually goes under texmf-local. That’s assuming you’re using the Minimals, of course. There are certain conventions regarding file locations. Most files of third party modules belong under tex/context/third/${MODULE}/, scripts that are supposed to be called from mtxrun under ./scripts/context/lua/third/${MODULE}, but I don’t think that convention is enforced when doing file lookups. – Philipp Gesang Dec 27 '14 at 11:36

Here’s a suggestion for debugging path lookup: First determine where Context stores its file info. Usually this is a directory named luatex-cache/context under the root texmf-cache or texmf-var. Then create a dummy file with a unique name somewhere under your texmf-local/. Now run mtxrun --generate and grep the luatex-cache recursively for the name of the dummy file. If the file was found it will show up in a description of tree contents (luatex-cache/context//trees/.lua). If not, then the location isn’t traversed by Context at all. – Philipp Gesang Dec 27 '14 at 11:44

Btw. one of my packages works in Context, Plain, and Latex: bitbucket.org/phg/enigma/src -- I just verified that placing the files in the respective trees under the texmf-local works just fine. – Philipp Gesang Dec 27 '14 at 11:55

@phg thanks for your patience and time! That solved it! I placed the file into /home/ludewich/tl2014/texmf-dist/tex/generic/pgf/math/foo/ba‌​r, ran mtxrun --generate and it was indexed properly and my minimal ran. I thought I had done the same after your first comment, but perhaps I made some mistake? If you write your hint regarding mtxrun into an answer, I will happily accept it. – Christian Feuersänger Dec 27 '14 at 14:10