10

Some time ago I tried to implement the \pdfadjustinterwordglue from pdfTeX in LuaTeX using a callback. Getting the callback to work is pretty simple, but there is another major problem in the way which I was so far not able to solve. The culprit is in the way parameters for spacing are set, which is for example

\knbscode\font`r=1100

This is a regular TeX assignment and this can be implemented in Lua using

\def\knbscode{\directlua{
local fid = token.scan_int()
local chr = token.scan_int()
local str = token.scan_string()

knbscode[fid] = knbscode[fid] or {}
knbscode[fid][chr] = knbscode[fid][chr] or {}
local int = string.gsub(str,"=","")
knbscode[fid][chr][field] = tonumber(int)
}\fontid}

That is to say, I look ahead for an integer, which is the font ID, another integer which is a character, and “string”, which according the LuaTeX manual is

returns a string given between {}, as \macro or as sequence of characters with catcode 11 or 12

The mindful reader might already notice a problem here because \knbscode\font`r=1100abc should parse by the normal TeX rules but will fail in this case. Luckily the usage of register assignment is sort of regular in TeX and such a situation rarely occurs.

The next big problem is that registers are not write-only. They can also be read using:

\the\knbscode\font`r

Of course, in this case, \knbscode should still scan for the font ID and the character but not for the assignment. My idea to solve this was to overload \the to peek at the next token and if it is \knbscode, delegate to another function. On the Lua end this looks like this

local t = lua.get_functions_table()
t[1] = function()
    local next = token.get_next()
    if next.csname == "knbscode" then
        token.put_next(token.create("theknbscode"))
    elseif next.csname == "stbscode" then
        token.put_next(token.create("thestbscode"))
    elseif next.csname == "shbscode" then
        token.put_next(token.create("theshbscode"))
    else
        token.put_next{ token.create("normalthe"), next }
    end
end

and on the TeX end I define \the to refer to this Lua function

\luadef\the1

such that it can expand within a single step. Except that it doesn't. My new \the does not expand within a single step because instead of putting the number from the register in the input stream I put a token which has to be expanded again.

Maybe this approach is just wrong altogether, so the question is relatively general:

How do I define a new TeX register with arguments in LuaTeX?

I actually have asked the same question some time ago on the LuaTeX mailing list with a simplified example, so I just got the answer: “well, use a normal register, duh” (https://tug.org/pipermail/luatex/2019-January/007040.html).


Below I have a full MWE with which you can play around.

\documentclass{article}
\usepackage{luacode}

\newcount\pdfadjustinterwordglue

\begin{luacode}
local subtypes = node.subtypes("glue")

local knbscode = {}

local t = lua.get_functions_table()
t[1] = function()
    local next = token.get_next()
    if next.csname == "knbscode" then
        token.put_next(token.create("theknbscode"))
    elseif next.csname == "stbscode" then
        token.put_next(token.create("thestbscode"))
    elseif next.csname == "shbscode" then
        token.put_next(token.create("theshbscode"))
    else
        token.put_next{ token.create("normalthe"), next }
    end
end

function set_knbscode(field)
    local fid = token.scan_int()
    local chr = token.scan_int()
    local str = token.scan_string()

    knbscode[fid] = knbscode[fid] or {}
    knbscode[fid][chr] = knbscode[fid][chr] or {}
    local int = string.gsub(str,"=","")
    knbscode[fid][chr][field] = tonumber(int)
end

function get_knbscode(field)
    local fid = token.scan_int()
    local chr = token.scan_int()

    knbscode[fid] = knbscode[fid] or {}
    knbscode[fid][chr] = knbscode[fid][chr] or {}
    tex.sprint([[\numexpr]] .. (knbscode[fid][chr][field] or 0) .. [[\relax]])
end

local microtype_spacing = function(head, tail)
    head, tail, success = node.kerning(head, tail)
    if tex.count.pdfadjustinterwordglue > 0 then
        for space in node.traverse_id(node.id("glue"), head) do
            if subtypes[space.subtype] == "spaceskip" then
                local prev = node.prev(space)
                if prev.id == node.id("glyph") then
                    local knbs = knbscode[prev.font]
                    if knbs and knbs[prev.char] then
                        local f = font.getfont(prev.font)
                        local em = f.parameters.quad

                        local width = knbs[prev.char].width or 0
                        local stretch = knbs[prev.char].stretch or 0
                        local shrink = knbs[prev.char].shrink or 0

                        local glue = node.new(node.id("glue"))
                        glue.width = width*em/1000
                        glue.stretch = stretch*em/1000
                        glue.shrink = shrink*em/1000

                        head = node.insert_before(head, space, glue)
                    end
                end
            end
        end
    end
end

luatexbase.add_to_callback("kerning", microtype_spacing, "microtype_spacing")
\end{luacode}

\protected\def\knbscode{\directlua{set_knbscode("width")}\fontid}
\protected\def\stbscode{\directlua{set_knbscode("stretch")}\fontid}
\protected\def\shbscode{\directlua{set_knbscode("shrink")}\fontid}

\let\normalthe\the

\def\theknbscode{\normalthe\directlua{get_knbscode("width")}\fontid}
\def\thestbscode{\normalthe\directlua{get_knbscode("stretch")}\fontid}
\def\theshbscode{\normalthe\directlua{get_knbscode("shrink")}\fontid}

\luadef\the1

\pdfadjustinterwordglue=1

% Bogus values, just for demonstration
\knbscode\font`r=1100
\stbscode\font`r=10
\shbscode\font`r=10

\knbscode\font`r 1100
\stbscode\font`r 10
\shbscode\font`r 10

\begin{document}

\the\knbscode\font`r
\the\stbscode\font`r
\the\shbscode\font`r

\input lorem

\end{document}
  • 2
    Apart from the theoretical point if it is possible: I don't think that it is really needed to try to emulate the low-level interface. Offer a sensible interface to set and get the values and ask microtype to use it. – Ulrike Fischer Jun 30 at 9:50
5

tl;dr: I do not think defining TeX-like registers from Lua is possible, but you can get a bit closer.

Fixing \the is easy: You only have to trigger one level of expansion form Lua. You could try token.expand but that function has a unconventional understanding which token to expand. So it's easier to rely on the standard TeX tool to get one level of expansion: \expandafter: You can insert \expandafter\relax into TeX's input stream, then read the \relax back with expansion.

But there are other problems. Let's assume that we want a "g" to use the same value as a "f". In pdfTeX we could say

\knbscode\font`g=\knbscode\font`f

This wouldn't work with emulated registers like this. This is because of two problems:

Setting \knbscode does only accept strings Lua con convert into numbers and not general TeX numbers and \knbscode works with \the, but still isn't an internal TeX number which can be used in all places where TeX accepts numbers. The first problem can be fixed: To read an optional = followed by an integer, use

token.scan_keyword'=' -- Scans an optional "="
local int = token.scan_int()

The second problem is harder to fix. If you want your register-like Lua value to be usable at all places which accept TeX numbers, you would need some way to determine from Lua if TeX tries to read a register or not. This information isn't available, so I do not think that perfectly emulating a register is possible.

Of course you can say that it's nice be be at least similar, but sometimes having a slightly different interface for the Lua version is better than to have subtly, much harder to detect, differences in usage.

The suggested changes lead to this version:

\documentclass{article}
\usepackage{luacode}

\newcount\pdfadjustinterwordglue

\begin{luacode}

local subtypes = node.subtypes("glue")

local knbscode = {}

-- Let's mmake sure that no one messes with our primitives:
local function frozentok(name)
  local tok = token.create(name)
  return token.new(tok.mode, tok.command)
end
local the, expandafter, relax = frozentok'the', frozentok'expandafter', frozentok'relax'
local t = lua.get_functions_table()
t[1] = function()
    local next = token.get_next()
    if next.csname == "knbscode" then
        token.put_next(token.create("theknbscode"))
    elseif next.csname == "stbscode" then
        token.put_next(token.create("thestbscode"))
    elseif next.csname == "shbscode" then
        token.put_next(token.create("theshbscode"))
    else
        token.put_next{ next }
    end
    token.put_next{ expandafter, relax, the }
    token.scan_token() -- The scanned token will always be the relax we just wrote, but expandafter does it's magic in the progress.
end

function set_knbscode(field)
    local fid = token.scan_int()
    local chr = token.scan_int()
    token.scan_keyword'='
    local int = token.scan_int()

    knbscode[fid] = knbscode[fid] or {}
    knbscode[fid][chr] = knbscode[fid][chr] or {}
    knbscode[fid][chr][field] = int
end

function get_knbscode(field)
    local fid = token.scan_int()
    local chr = token.scan_int()

    knbscode[fid] = knbscode[fid] or {}
    knbscode[fid][chr] = knbscode[fid][chr] or {}
    tex.sprint([[\numexpr]] .. (knbscode[fid][chr][field] or 0) .. [[\relax]])
end

local microtype_spacing = function(head, tail)
    head, tail, success = node.kerning(head, tail)
    if tex.count.pdfadjustinterwordglue > 0 then
        for space in node.traverse_id(node.id("glue"), head) do
            if subtypes[space.subtype] == "spaceskip" then
                local prev = node.prev(space)
                if prev.id == node.id("glyph") then
                    local knbs = knbscode[prev.font]
                    if knbs and knbs[prev.char] then
                        local f = font.getfont(prev.font)
                        local em = f.parameters.quad

                        local width = knbs[prev.char].width or 0
                        local stretch = knbs[prev.char].stretch or 0
                        local shrink = knbs[prev.char].shrink or 0

                        local glue = node.new(node.id("glue"))
                        glue.width = width*em/1000
                        glue.stretch = stretch*em/1000
                        glue.shrink = shrink*em/1000

                        head = node.insert_before(head, space, glue)
                    end
                end
            end
        end
    end
end

luatexbase.add_to_callback("kerning", microtype_spacing, "microtype_spacing")

\end{luacode}

\protected\def\knbscode{\directlua{set_knbscode("width")}\fontid}
\protected\def\stbscode{\directlua{set_knbscode("stretch")}\fontid}
\protected\def\shbscode{\directlua{set_knbscode("shrink")}\fontid}

\def\theknbscode{\directlua{get_knbscode("width")}\fontid}
\def\thestbscode{\directlua{get_knbscode("stretch")}\fontid}
\def\theshbscode{\directlua{get_knbscode("shrink")}\fontid}

\luadef\the1

\pdfadjustinterwordglue=1

% Bogus values, just for demonstration
\knbscode\font`r=1100
\stbscode\font`r=10
\shbscode\font`r=10

\knbscode\font`r 1100
\stbscode\font`r 10
\shbscode\font`r 10

\begin{document}

\the\knbscode\font`r
\the\stbscode\font`r
\the\shbscode\font`r

\input lorem

\end{document}
  • I've had a look in the source. It seems that to support this in the engine, we'd need Lua hooks into the case assign_font_int_cmd: in prefixed_command and scan_something_internal. – Henri Menke Jul 1 at 5:09
  • @HenriMenke You certainly have to patch scan_something_internal but instead of reusing the font commands you probably would add new commands. Then you might not even need to use prefixed_command: The only prefix that applies to registers is \global and Lua registers in most cases will not know about local/global settings(Like \knbscode). – Marcel Krüger Jul 1 at 9:16
  • It should be a pretty straight forward patch, the question is if there is any sufficiently good use case to warrent including it into the engine. – Marcel Krüger Jul 1 at 9:18
  • @MarcelKrüger What criteria are used for adding something to the LuaTeX engine? – Faheem Mitha Aug 15 at 6:25
  • @FaheemMitha In general I never found a real pattern in what they accept and don't accept, but currently LuaTeX is pretty much frozen, so new features are generally rejected. That doesn't mean that there can't be any exceptions though, so you can try... – Marcel Krüger Aug 15 at 8:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.