I am trying to debug a much larger problem, but the smoking gun is some (apparently) strange behavior with LuaTeX’s fontloader library.

Minimal Example

Here is file backmap.tex:

\directlua { tex.enableprimitives('',tex.extraprimitives()) }
\directlua { dofile('define_font.lua') }

Here is file define_font.lua:

function read_font (name, size, fontid)
  -- Load file using fontloader.open
  local f = fontloader.open (name)
  local fonttable = fontloader.to_table(f)

  for char, glyph in pairs(fonttable.map.map) do
    if char == 0x0020      -- SPACE
    or char == 0x110300    -- Some ‘character’ in Amiri font
    or char == 0x06A9 then -- ARABIC LETTER KEHEH
      local backmap = fonttable.map.backmap[glyph]      
      texio.write_nl(string.format("\n\nGLYPH: 0x%x, CHAR: 0x%x, BACKMAP: 0x%x", glyph, char, backmap))
      if char ~= backmap then
        texio.write_nl("char and backmap value DIFFERENT")
        texio.write_nl(string.format("fonttable.map.map[0x%x]: 0x%x", char, fonttable.map.map[char]))
        texio.write_nl(string.format("fonttable.map.map[0x%x]: 0x%x", char, fonttable.map.map[backmap]))
        texio.write_nl("char and backmap value SAME")

  return { }

-- Register OpenType font loader in define_font callback.
callback.register('define_font', read_font, "font loader")

And here is the output of running luatex --fmt=plain backmap.tex:

This is LuaTeX, Version beta-0.87.1 (TeX Live 2016/dev)

GLYPH: 0x3, CHAR: 0x20, BACKMAP: 0xa0
char and backmap value DIFFERENT
fonttable.map.map[0x20]: 0x3
fonttable.map.map[0xa0]: 0x3

GLYPH: 0x206, CHAR: 0x6a9, BACKMAP: 0x6a9
char and backmap value SAME

GLYPH: 0x987, CHAR: 0x110300, BACKMAP: 0x110300
char and backmap value SAME
! error  (font): lua-loaded font '51' has no name!
!  ==> Fatal error occurred, bad output DVI file produced!
No pages of output.


I am trying to read all the characters and glyphs from fonttable.map.map, and then comparing the values of the char to the corresponding value of fonttable.map.backmap[glyph].

My expectation is that the fonttable.map.backmap table should map the glyph back to the same character, that maps to that specific glyph. But as you can see from the output above, the backmap for the glyph corresponding to U+0020 (SPACE), instead maps to U+00A0 (NO-BREAK SPACE).

The reason this is causing me problems is that I am using an external shaping engine (Harfbuzz) to get glyphs for characters in a paragraph, and then trying to check the backmap to see if a glyph maps back to U+0020 (SPACE) character, and then intercept it to add a glue node instead.

My questions:

  1. Is it valid for a font to be that way, i.e. backmap does not match glyph back to the same character?
  2. Any suggestions for how I can intercept spaces in general? Should I look for any kind of space and substitute it for glue instead?
  1. This is fine, the returned codepoints are in PUA (private use area), because they should be displayed with other glyphs than those used in the input file. This happens for non-standard ligatures, with various scripts such as Arabic, and other glyphs produced by the shaping process which aren't defined as normal Unicode characters.

  2. you can also shape just words and leave original glues in the node list. This way you will preserve non breaking and other types of spaces inserted by the user. The other way is to use table with Unicode character info to detect various space types and act accordingly to it. But when all spaces produced by the shaper are non-beaking, it doesn't seem too useful to me. Especially in the case of Urdu, where we don't have functional hyphenation in LuaTeX

There is an example to illustrate the issue of various spaces:

local glue_id = node.id "glue"
local glyph_id = node.id "glyph"
local penalty_id = node.id "penalty"
local kern_id = node.id "kern"
local utfchar = unicode.utf8.char
local onept = tex.sp "1pt"
local function clb(head)
  local t = {}
  local function printword()
    t = {}
  for n in node.traverse(head) do
    if n.id == glyph_id then
      t[#t+1] = utfchar(n.char)
    elseif n.id == glue_id then
      local spec = n.spec
      local width = spec.width / onept
      local stretch = spec.stretch / onept
      local shrink = spec.shrink /onept
      print("glue type: ",n.subtype, width, stretch, shrink)
    elseif n.id == penalty_id then
      print("penalty", n.penalty / onept)
    elseif n.id == kern_id and n.subtype == 1 then 
      print("kern", n.kern / onept)
  return head
luatexbase.add_to_callback("pre_linebreak_filter", clb, "print spaces")

hello world~there\,are\quad various\qquad space\ types


this produces this output in the console:

glue type:      0       3.3333282470703 1.6666564941406 1.1111145019531
penalty 0.152587890625
glue type:      0       3.3333282470703 1.6666564941406 1.1111145019531
kern    1.6667175292969
glue type:      0       10.000015258789 0       0
glue type:      0       20.000030517578 0       0
glue type:      0       3.3333282470703 1.6666564941406 1.1111145019531
penalty 0.152587890625
glue type:      15      0       1       0

to support these cases, I would save glue, penalty and kern nodes between words in a table and then insert them back at correct places. It might be difficult in BiDi, I guess.

  • So is it accurate to say – for complex fonts like for e.g. Arabic fonts, the fonttable.map.map is not merely a mapping of unicode characters to glyphs, but also of some additional codepoints (in the so-called PUA) to their corresponding glyphs as well? – vyom Dec 31 '15 at 17:38
  • About point 2 above – a) the problem with leaving original glues in the node list is that it prevents one from doing full paragraph BiDi shaping. This maybe required for scenarios where for example , an English paragraph contains a few Arabic words in between. b) In common scenarios where paragraphs are homogenous, i.e. are typeset using the same script, font and size, it is probably more efficient to shape them in one go, than to shape each individual word. – vyom Dec 31 '15 at 17:49
  • yes, glyphs which don't have normal Unicode character must be accessed using PUA – michal.h21 Dec 31 '15 at 18:01
  • @vyom I think that BiDi should be handled separately, direction nodes should be inserted before text shaping. promisingly looks this code from Khaled – michal.h21 Dec 31 '15 at 18:04
  • @vyom in Latin text, there are often various space types used - thin spaces, non-breaking spaces, etc. these are definitely cases which needs a attention – michal.h21 Dec 31 '15 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.