XeLaTeX, LuaLaTeX, fontspec, unicode and normalization

Question

I am troubled by the way LuaTeX and XeLaTeX normalize unicode composed character. I mean NFC / NFD.

See the following MWE

\documentclass{article}
\usepackage{fontspec}
\setmainfont{Linux Libertine O}

\begin{document}


ᾳ GREEK SMALL LETTER ALPHA (U+03B1) + COMBINING GREEK YPOGEGRAMMENI (U+0345)

ᾳ GREEK SMALL LETTER ALPHA WITH YPOGEGRAMMENI (U+1FB3)

\end{document}

With LuaLaTeX I obtain:

As you can see, Lua does not normalize unicode character, and as Linux Libertine has a bug (http://sourceforge.net/p/linuxlibertine/bugs/266/), I have a bad character.

With XeLaTeX, I obtain

As you can see, the Unicode is normalized.

My three questions are :

1. Why XeLaTeX has normalized (in NFC), despite I have not used \XeTeXinputnormalization
2. Did this feature change from the past. Because my previous, with TeXLive 2012 send be a bad result (see the articles I wrote at this time http://geekographie.maieul.net/Normalisation-des-caracteres)
3. Does LuaTeX has option like there is \XeTeXinputnormalization in XeTeX?

Answer 1

I don't know the answer for first two questions, as I don't use XeTeX, but I want to provide option for the third question.

Thanks to Arthur's code I was able to create basic package for unicode normalization in LuaLaTeX. The code needed only slight modifications to work with current LuaTeX. I will post only main Lua file here, full project is available on Github as uninormalize.

Sample usage:

\documentclass{article}
\usepackage{fontspec}
\usepackage[czech]{babel}
\setmainfont{Linux Libertine O}
\usepackage[nodes,buffer=false, debug]{uninormalize}
\begin{document}

Some tests:
\begin{itemize}
  \item combined letter ᾳ %GREEK SMALL LETTER ALPHA (U+03B1) + COMBINING GREEK YPOGEGRAMMENI (U+0345)
  \item normal letter ᾳ% GREEK SMALL LETTER ALPHA WITH YPOGEGRAMMENI (U+1FB3)
\end{itemize}

Some more combined and normal letters: 
óóōōöö

Linux Libertine does support some combined chars: \parbox{4em}{příliš}
\end{document}

(note that correct version of this file is on Github, combined letters were transferred incorrectly in this example)

Main idea of the package is following: process the input, and when letter followed by combined marks is found, then it is replaced by normalized NFC form. Two methods are provided, my first approach was to use node processing callbacks to replace decomposed glyphs with normalized characters. This would have a advantage in that it would be possible to switch on and off the processing anywhere, using node attributes. The other possible feature could be checking if the current font contains normalized character and use original form if it doesn't. Unfortunately, in my tests it fails with some characters, notably composed í is in the nodes as dotless i + ´, instead of i + ´, which after the normalization doesn't produce the correct character, so composed chars are used instead. But this produce output with bad placing of the accent. So this method needs either some correction, or it is totally wrong.

So the other method is to use process_input_buffer callback to normalize the input file as it is read from the disk. This method doesn't allow to use info from fonts, nor it allows to turning off in the middle of the line, but it is significantly easier to implement, the callback function may look like this:

function buffer_callback(line) 
  return NFC(line)
end 

which is really nice finding after three days spent on node processing version.

For curiosity this is the Lua package:

local M = {}
dofile("unicode-names.lua")
dofile('unicode-normalization.lua')
local NFC = unicode.conformance.toNFC
local char = unicode.utf8.char
local gmatch = unicode.utf8.gmatch
local name = unicode.conformance.name
local byte = unicode.utf8.byte
local unidata = characters.data
local length = unicode.utf8.len

M.debug = false

-- for some reason variable number of arguments doesn't work
local function debug_msg(a,b,c,d,e,f,g,h,i)
  if M.debug then
    local t = {a,b,c,d,e,f,g,h,i}
    print("[uninormalize]", unpack(t))
  end
end

local function make_hash (t) 
  local y = {}
  for _,v in ipairs(t) do 
    y[v] = true
  end
  return y
end

local letter_categories = make_hash {"lu","ll","lt","lo","lm"}

local mark_categories = make_hash {"mn","mc","me"}

local function printchars(s)
    local t = {}
    for x in gmatch(s,".") do
        t[#t+1] = name(byte(x))
    end
    debug_msg("characters",table.concat(t,":"))
end

local categories = {}


local function get_category(charcode)
  local charcode = charcode or ""
  if categories[charcode] then
    return categories[charcode] 
  else
    local unidatacode = unidata[charcode] or {}
    local category = unidatacode.category
    categories[charcode] = category
    return category
  end
end

-- get glyph char and category
local function glyph_info(n)
  local char = n.char
  return char, get_category(char)
end

local function get_mark(n)
  if n.id == 37 then
    local character, cat = glyph_info(n)
    if mark_categories[cat] then
      return char(character)
    end
  end
  return false
end

local function make_glyphs(head, nextn,s, lang, font, subtype) 
  local g = function(a) 
    local new_n = node.new(37, subtype)
    new_n.lang = lang
    new_n.font = font
    new_n.char = byte(a)
    return new_n
  end
  if length(s) == 1 then
    return node.insert_before(head, nextn,g(s))
  else
    local t = {}
    local first = true
    for x in gmatch(s,".") do
      debug_msg("multi letter",x)
        head, newn = node.insert_before(head, nextn, g(x))
    end
    return head
  end
end

local function normalize_marks(head, n)
  local lang, font, subtype = n.lang, n.font, n.subtype
  local text = {}
  text[#text+1] = char(n.char)
  local head, nextn = node.remove(head, n)
  --local nextn = n.next
  local info = get_mark(nextn)
  while(info) do
    text[#text+1] = info
    head, nextn = node.remove(head,nextn)
    info = get_mark(nextn)
  end
  local s = NFC(table.concat(text))
  debug_msg("We've got mark: " .. s)
  local new_n = node.new(37, subtype)
  new_n.lang = lang
  new_n.font = font
  new_n.char = byte(s)
  --head, new_n = node.insert_before(head, nextn, new_n)
  -- head, new_n = node.insert_before(head, nextn, make_glyphs(s, lang, font, subtype))
  head, new_n = make_glyphs(head, nextn, s, lang, font, subtype)
  local t = {}
  for x in node.traverse_id(37,head) do
    t[#t+1] = char(x.char)
  end
  debug_msg("Variables ", table.concat(t,":"), table.concat(text,";"), char(byte(s)),length(s))
  return head, nextn
end

local function normalize_glyphs(head, n)
  --local charcode = n.char
  --local category = get_category(charcode)
  local charcode, category = glyph_info(n)
  if letter_categories[category] then 
    local nextn = n.next
    if nextn.id == 37 then
      --local nextchar = nextn.char
      --local nextcat = get_category(nextchar)
      local nextchar, nextcat = glyph_info(nextn)
      if mark_categories[nextcat] then
        return normalize_marks(head,n)
      end
    end
  end
  return head, n.next 
end


function M.nodes(head)
    local t = {}
    local text = false
  local n = head
    -- for n in node.traverse(head) do
  while n do
        if n.id == 37 then
      local charcode = n.char
            debug_msg("unicode name",name(charcode))
            debug_msg("character category",get_category(charcode))
            t[#t+1]= char(charcode)
            text = true
      head, n = normalize_glyphs(head, n)
        else
            if text then
                local s = table.concat(t)
                debug_msg("text chunk",s)
                --printchars(NFC(s))
                debug_msg("----------")
            end
            text = false
            t = {}
      n = n.next
        end
    end
    return head
end

--[[
-- These functions aren't needed when processing buffer. We can call NFC on the whole input line
local unibytes = {}

local function get_charcategory(s)
  local s = s or ""
  local b = unibytes[s] or byte(s) or 0
  unibytes[s] = b
  return get_category(b)
end

local function normalize_charmarks(t,i)
  local c = {t[i]}
  local i = i + 1
  local s = get_charcategory(t[i])
  while mark_categories[s] do
    c[#c+1] = t[i]
    i = i + 1
    s = get_charcategory(t[i])
  end
  return NFC(table.concat(c)), i
end

local function normalize_char(t,i)
  local ch = t[i]
  local c = get_charcategory(ch)
  if letter_categories[c] then
    local nextc = get_charcategory(t[i+1])
    if mark_categories[nextc] then
      return normalize_charmarks(t,i)
    end
  end
  return ch, i+1
end
-- ]]
function M.buffer(line)
  --[[
  local t = {}
  local new_t = {}
  -- we need to make table witl all uni chars on the line
  for x in gmatch(line,".") do
    t[#t+1] = x
  end
  local i = 1
  -- normalize next char
  local c, i = normalize_char(t, i)
  new_t[#new_t+1] = c
  while t[i] do
    c, i = normalize_char(t,i)
    -- local  c = t[i]
    -- i =  i + 1
    new_t[#new_t+1] = c
  end
  return table.concat(new_t)
  --]]
  return NFC(line)
end


return M

and now is the time for some pictures.

without normalization:

you can see that composed Greek char is wrong, other combinations are supported by Linux Libertine

with node normalization:

Greek letters are correct, but í in first příliš is wrong. this is the issue I was talking about.

and now the buffer normalization:

everything is alright now