# Is direct utf8 input of combining diacritics in math mode possible with lualatex?

I am trying to typeset a document with combining diacritics directly input in it. I use LuaLaTex. Here is a minimal example illustrating the original issue:

\documentclass{minimal}
\usepackage{unicode-math}
\setmathfont{XITS Math}
\begin{document}
$v⃗$
\end{document}

The above vector arrow (U+20D7) is completely lost in the output. In text mode it would be shown, but in math mode it was discarded from the horizontal list alltogether.

Then I tried the following:

\documentclass{minimal}
\usepackage{unicode-math}
\setmathfont{XITS Math}
{
\catcode\_=11\relax
\catcode\:=11\relax
\gdef\SetMathCode#1#2{\um_set_mathcode:nnn{#1}{#2}\um_symfont_tl}
}
\SetMathCode{"20D7}\mathaccent
\begin{document}
$v⃗$
\end{document}

This code essentially uses \Umathcode, indirectly through a macro in the unicode-math package. The reason is, that I found I had to change the math family of the arrow to the XITS font. The mapping of the diacritics (and possibly some of the other characters) are not set up automatically for math mode.

Now the arrow is typeset adjacently to the right of the accented v. I want it to be typeset as accent, above the v. The \vec macro and \Umathaccent work, but I want to make the formulas plain-text readable if possible. (I use the Emacs quail system for input.)

My LuaTeX version is beta-0.70.2, TeX Live 2012, LaTeX2e <2011/06/27>

XITS font is version 1.105.

Note:

Obviously the problem arises when accenting the special script-like letters.

In the end, the issue seems to be with the handling of the special top-accent glyph metric. It is supposed to be done as described in the "Math accent handling" section of the luatex manual, but in reality is done only for the \Umathaccent command, and I think forgotten for combining characters. The text version of the font uses some other mechanism with horizontal offsets (called "bearings"?), and goes around this limitation.

I will investigate this a bit further. If it is a core issue, I should file this with the LuaTeX guys. Consider the question closed. It became too specific anyway.

-
I can reproduce this. The log file also has the line Missing character: There is no ⃗ (U+20D7) in font cmmi10! –  Martin Schröder Oct 14 '12 at 23:29
Yes. In the first case, the arrow symbol is mapped to the default font and TeX complains. After I remap it in the second example, TeX does not complain anymore, but prints the arrow to the right of the main glyph. –  simeonz Oct 15 '12 at 6:06
same with xetex. As an aside when I copy your $v⃗$ to my emacs buffer (unicode encoded) the arrow appears to the right of v, so these are really two characters. Is there a unicode glyph "v with arrow" that you could put into your source? –  jfbu Oct 19 '12 at 9:12
When I copy it back and paste it, it show ok on my end. I suppose it depends on the font you use. I have configured STIX as the font for Emacs, so it shows. Btw, I just found out something by experiment and added a note above. –  simeonz Oct 19 '12 at 19:25
IIRC, this issue was discussed in the context mailing list and it was decided not to add this behavior as text diactrics and math accents are different things. –  Aditya Dec 9 '13 at 18:05

I got it working with a lua script. Your minimal example becomes:

\documentclass{minimal}
\usepackage{unicode-math}
\setmathfont{XITS Math}
\AtBeginDocument{\directlua{require("combining_preprocessor.lua")}}
\newcommand{\⃗}[1]{\ensuremath{\vec{#1}}}
\begin{document}
$v⃗$
\end{document}

The idea is that it's difficult to make LaTeX handle a command or macro that comes after its argument, which is how Unicode combining characters work, so we use would like a preprocessor to move the accent so it comes before its argument. That is, map v⃗ to \⃗{v} in a script, and then define whatever action you want \⃗ to have. (That's a backslash followed by a combining arrow, which should be printed above the backslash.)

My lua script does most (all?) of the combining characters, so you just need to define what they should do in the .tex file. Many accents on the same character is possible. Example:

\documentclass{minimal}

\usepackage{unicode-math}
\setmathfont{XITS Math}

\AtBeginDocument{\directlua{require("combining_preprocessor.lua")}}

\newcommand{\̂}[1]{\ensuremath{\hat{#1}}}
\newcommand{\⃑}[1]{\ensuremath{\vec{#1}}}
\newcommand{\̱}[1]{\ensuremath{\underline{#1}}}
\newcommand{\́}[1]{\ensuremath{\acute{#1}}}

\usepackage{stackrel}
\newcommand{\᷽}[1]{\ensuremath{\stackrel[\approx]{}{#1}}}

\begin{document}

Hello

$ℂ̂$ is hat on $ℂ$, more on $ℂ̂⃑$ (stress test)

$ℂ̂ x̂$

Many combining accents on $x᷽̱̂́⃑$ is cool.

\end{document}

(My browser doesn't do the many combining characters justice here, but it looks nice in the PDF file.)

Not sure if this is the ideal way of doing things, but for what it's worth, here is combining_preprocessor.lua:

function minornil(a, b)
if a == nil and b == nil then
return nil
elseif a == nil then
return b
elseif b == nil then
return a
else
return math.min(a, b)
end
end

function findfirstcombining(line, n)
local a = string.find(line, "\204[\128-\191]", n)     -- From U0300,
local b = string.find(line, "\205[\128-\175]", n)     -- to U036F.
a = minornil(a, b)
b = string.find(line, "\226\131[\144-\176]", n) -- U20D0 to U20F0
a = minornil(a, b)
b = string.find(line, "\225\183[\128-\191]", n) -- U1DC0 to U1DFF
a = minornil(a, b)
return a
end

function is_utf8_continuation(byte)
return byte < 191 and byte > 127
end

function find_next_utf8_char(str, n)
while str:byte(n) ~= nil and is_utf8_continuation(str:byte(n)) do
n = n + 1
end
return n
end

function combining_iter(str)
local n = 0
return function ()
n = (n ~= nil) and findfirstcombining(str, n + 1)
return n
end
end

function dobuffer(line)
local n1 = 0
local t = {}
for n2 in combining_iter(line) do
if n2 > n1 then
local n3 = n2
repeat
n3 = n3 - 1
until not is_utf8_continuation(line:byte(n3))
table.insert(t, string.sub(line, n1, n3 - 1))
n1 = find_next_utf8_char(line, n2 + 1)
local comb = {}
table.insert(comb, "\\" .. string.sub(line, n2, n1 - 1) .. "{")
table.insert(comb, string.sub(line, n3, n2 - 1) .. "}")
n2 = findfirstcombining(line, n1)
while n2 == n1 do
n1 = find_next_utf8_char(line, n2 + 1)
table.insert(comb, 1, "\\" .. line:sub(n2, n1 - 1) .. "{")
table.insert(comb, "}")
n2 = findfirstcombining(line, n1)
end
table.insert(t, table.concat(comb))
end
end
table.insert(t, string.sub(line, n1))
print(table.concat(t))
return table.concat(t)
end

dobuffer, "combining_preprocessor", 1)
-
Welcome to TeX.SX! –  Claudio Fiandrino Dec 9 '13 at 18:26

unicode-math does not set \mathcode for Unicode accents the same it does with other Unicode characters like math italics, so TeX looks for them in the first math font which is Computer Modern Math Italic (cmmi10 in the log) which does not have the accents (not in the Unicode positions at least).

But even if unicode-math did set the \mathcode the math accent will not be positioned properly (as you already noted), because accents must be called with \(U|XeTeX)mathaacent primitive for TeX to do its math accent positioning magic.

It might be possible to make the accents active math characters and map them to the respective macros (unicode-math already does this sort of tricky to allow direct input of other Unicode characters), but this is left as an exercise to the reader (read: I don’t know how to do this and last time I tried to understand that code I was on the verge of losing my sanity).

The engine itself knows nothing about Unicode characters, it the responsibility of the user (or macro package writer) to tell it which character is to be treated as an accent or a big operator or an opening symbol etc. using the appropriate primitive and/or math code (otherwise things would be very inflexible).

-
I understand your point. Indeed, you can have two consecutive characters in the input stream, both with a "top accent" value set for them in the Open Type font. Depending on the intent, the second one may have to be the accent, the first may have to be accent and the second a base glyph, or both may have to be stacked as accents on the preceding character. However, I thought that "combining diacritical marks" is a very straightforward category of Unicode code points. I don't believe there is flexibility to their interpretation when encountered in an input stream. –  simeonz Oct 21 '12 at 14:48
I mean, when you open utf8 text with any (Unicode aware) text editor, there is a standard-conforming display of the input stream, so to speak, or am I wrong? Should LuaTeX not choose this interpretation of the combining diacritical marks code points by default? Thanks for the answer. I will up-vote it, of course, but to accept it, I think we should clarify this bit for the sake of the community. –  simeonz Oct 21 '12 at 15:04
Regarding the active character approach, AFAIK, it has limitations. The accent code has to be prefix to the accented character's code. With simple accents, like prime, it can be done independently from the base glyph. Then the active character becomes independent, so to speak, and needs no arguments, but this differs fundamentally from the \Umathaccent style of handling. Thanks for the response again. –  simeonz Oct 21 '12 at 15:11
After I gave your answer some additional thought, I concluded I may have misunderstood what you meant - that the code points have configurable properties and LuaTeX does not understand the Unicode layout at all. Then, the question remains - can the package writer request a "combining character" behavior for a code point. Say with a primitive, or even by Lua script. –  simeonz Oct 21 '12 at 16:16
Indeed, LuaTeX (or XeTeX or even TeX, for that matter) don’t know/hard code any character properties, not even that \ is used for escaping or {} for grouping, it is all supplied by macro packages (or in the format). The same for math characters, you have to tell the engine of what category they are for proper spacing and placement to take place. –  Khaled Hosny Oct 21 '12 at 16:25