Strange output with LuaTeX’s fontloader library

Question

I am trying to debug a much larger problem, but the smoking gun is some (apparently) strange behavior with LuaTeX’s fontloader library.

Minimal Example

Here is file backmap.tex:

\directlua { tex.enableprimitives('',tex.extraprimitives()) }
\directlua { dofile('define_font.lua') }
\font\noto={../../fonts/amiri-regular.ttf}
\bye

Here is file define_font.lua:

function read_font (name, size, fontid)
  -- Load file using fontloader.open
  local f = fontloader.open (name)
  local fonttable = fontloader.to_table(f)
  fontloader.close(f)

  for char, glyph in pairs(fonttable.map.map) do
    if char == 0x0020      -- SPACE
    or char == 0x110300    -- Some ‘character’ in Amiri font
    or char == 0x06A9 then -- ARABIC LETTER KEHEH
      local backmap = fonttable.map.backmap[glyph]      
      texio.write_nl(string.format("\n\nGLYPH: 0x%x, CHAR: 0x%x, BACKMAP: 0x%x", glyph, char, backmap))
      if char ~= backmap then
        texio.write_nl("char and backmap value DIFFERENT")
        texio.write_nl(string.format("fonttable.map.map[0x%x]: 0x%x", char, fonttable.map.map[char]))
        texio.write_nl(string.format("fonttable.map.map[0x%x]: 0x%x", char, fonttable.map.map[backmap]))
      else
        texio.write_nl("char and backmap value SAME")
      end
    end
  end

  return { }
end

-- Register OpenType font loader in define_font callback.
callback.register('define_font', read_font, "font loader")

And here is the output of running luatex --fmt=plain backmap.tex:

This is LuaTeX, Version beta-0.87.1 (TeX Live 2016/dev)
(./backmap.tex


GLYPH: 0x3, CHAR: 0x20, BACKMAP: 0xa0
char and backmap value DIFFERENT
fonttable.map.map[0x20]: 0x3
fonttable.map.map[0xa0]: 0x3


GLYPH: 0x206, CHAR: 0x6a9, BACKMAP: 0x6a9
char and backmap value SAME


GLYPH: 0x987, CHAR: 0x110300, BACKMAP: 0x110300
char and backmap value SAME
! error  (font): lua-loaded font '51' has no name!
!  ==> Fatal error occurred, bad output DVI file produced!
No pages of output.
(

Explanation

I am trying to read all the characters and glyphs from fonttable.map.map, and then comparing the values of the char to the corresponding value of fonttable.map.backmap[glyph].

My expectation is that the fonttable.map.backmap table should map the glyph back to the same character, that maps to that specific glyph. But as you can see from the output above, the backmap for the glyph corresponding to U+0020 (SPACE), instead maps to U+00A0 (NO-BREAK SPACE).

The reason this is causing me problems is that I am using an external shaping engine (Harfbuzz) to get glyphs for characters in a paragraph, and then trying to check the backmap to see if a glyph maps back to U+0020 (SPACE) character, and then intercept it to add a glue node instead.

My questions:

1. Is it valid for a font to be that way, i.e. backmap does not match glyph back to the same character?
2. Any suggestions for how I can intercept spaces in general? Should I look for any kind of space and substitute it for glue instead?

Answer 1

1. This is fine, the returned codepoints are in PUA (private use area), because they should be displayed with other glyphs than those used in the input file. This happens for non-standard ligatures, with various scripts such as Arabic, and other glyphs produced by the shaping process which aren't defined as normal Unicode characters.

2. you can also shape just words and leave original glues in the node list. This way you will preserve non breaking and other types of spaces inserted by the user. The other way is to use table with Unicode character info to detect various space types and act accordingly to it. But when all spaces produced by the shaper are non-beaking, it doesn't seem too useful to me. Especially in the case of Urdu, where we don't have functional hyphenation in LuaTeX

There is an example to illustrate the issue of various spaces:

\documentclass{article}
\usepackage{luacode}
\begin{luacode*}
local glue_id = node.id "glue"
local glyph_id = node.id "glyph"
local penalty_id = node.id "penalty"
local kern_id = node.id "kern"
local utfchar = unicode.utf8.char
local onept = tex.sp "1pt"
local function clb(head)
  local t = {}
  local function printword()
    print(table.concat(t))
    t = {}
  end
  for n in node.traverse(head) do
    if n.id == glyph_id then
      t[#t+1] = utfchar(n.char)
    elseif n.id == glue_id then
      printword()
      local spec = n.spec
      local width = spec.width / onept
      local stretch = spec.stretch / onept
      local shrink = spec.shrink /onept
      print("glue type: ",n.subtype, width, stretch, shrink)
    elseif n.id == penalty_id then
      print("penalty", n.penalty / onept)
    elseif n.id == kern_id and n.subtype == 1 then 
      printword()
      print("kern", n.kern / onept)
    end
  end
  return head
end
luatexbase.add_to_callback("pre_linebreak_filter", clb, "print spaces")
\end{luacode*}
\begin{document}

hello world~there\,are\quad various\qquad space\ types

\end{document}

this produces this output in the console:

hello
glue type:      0       3.3333282470703 1.6666564941406 1.1111145019531
penalty 0.152587890625
world
glue type:      0       3.3333282470703 1.6666564941406 1.1111145019531
there
kern    1.6667175292969
are
glue type:      0       10.000015258789 0       0
various
glue type:      0       20.000030517578 0       0
space
glue type:      0       3.3333282470703 1.6666564941406 1.1111145019531
penalty 0.152587890625
types
glue type:      15      0       1       0

to support these cases, I would save glue, penalty and kern nodes between words in a table and then insert them back at correct places. It might be difficult in BiDi, I guess.