Ligature suppression algorithm fails if the word in question contains an Umlaut (dieresis) before the ligature

This is a follow-up posting prompted by Taco Hoekwater's recent excellent answer to a question I posed a few months ago. It concerns the use of the ligature suppression algorithm for words (mainly, but not exclusively, German-language words) that contain Umlaute (diereses).

Taking Taco's MWE as the starting point (not reproduced below because it's quite long), add the instructions

\suppressligature{lffach}{lf|fach} %% no f-ligature in words containing the string "lffach"
\suppressligature{rflich}{rf|lich} %% ditto for words containing the string "rflich"
\suppressligature{mpfflug}{mpf|flug}


and the following six words (which all contain potential f-ligature glyphs):

elffach zwölffach                %% ff ligature to be suppressed
verwerflich dörflich             %% fl ligature to be suppressed
Kampfflugzeug Düsenkampfflugzeug %% ffl ligature to be replaced with f-fl


after \begin{document}. This results in the following output:

Notice that the three words which do not contain an Umlaut before the | ligature suppression point are treated correctly, i.e., the ff, fl, and ffl ligatures are broken up in the manner indicated by the applicable \suppressligature commands. In contrast, ligature suppression fails for the second set of three words -- apparently because they contain an Umlaut. This happens with words containing an ä Umlaut as well, by the way.

Furthermore, I've determined that this problem only appears to happen with words that contain an Umlaut before the ligature suppression point. Words that contain an Umlaut after the ligature suppression point indicated by a corresponding \suppressligature instruction are treated correctly. E.g., when supplying the instruction \suppressligature{uflös}{uf|lös}, the words auflösen and Auflösung are both rendered without the fl ligature.

A word about my TeX setup: I mostly run TeXLive2011 and use TeXWorks as my editor program. However, the problem also occurs under MiKTeX2.9, with all the latest updates installed, and with WinEdt7.0 as the editor. Both editors are fully unicode-aware.

Question: How does Taco's code need to be edited/modified to circumvent the problem(s) described in this posting?

-
I don't have a fix, but I assume this is due to the fact that string.find() (and unicode.utf8.find()) return the number of bytes, not chars. The debug message Position n is off by one in 'Zwölffach'. – topskip Jul 11 '12 at 20:58
even though you're apparently using unicode (utf8?) input, this looks suspiciously similar to the reasoning by which it's decided whether or not hyphenation is applied. what happens if you feed those same words into \showhyphens? – barbara beeton Jul 11 '12 at 20:58
You can easily check this if you compare elffach and ölffach with debug messages. They have the same amount of letters before/after the ligature and yet give different results on the console. – topskip Jul 11 '12 at 21:04
See the message here for a solution to utf8-find: tug.org/pipermail/luatex/2010-March/001262.html – topskip Jul 12 '12 at 5:11
@Patrick: you should convert one of your comments into an answer. The byte-based return value is exactly the issue. – Taco Hoekwater Jul 13 '12 at 7:31

The problem is that string.find returns a byte location, and in this case you want to have a character index when comparing nodes to strings. The solution is to replace

table.insert(throwliga,{match,n,k})


with

table.insert(throwliga,{prefix_length(f,match),n,k})


where prefix_length is defined like this:

  local prefix_length = function(word,byte)
return unicode.utf8.len(string.sub(word,0,byte))
end


Which simply returns the length in characters of the prefix of word that ends at byte.

Note: This change only fixes the current problem, it does not solve cases where there are unicode characters inside the strings in the noliga table. Please do not ask me to write that as well, there should be some challenge left ;)

-
Many thanks for providing this solution! Much appreciated! – Mico Jul 13 '12 at 16:41
Taco: I've finally posted a complete first version to GitHub of the package "selnolig", which provides ligature suppression methods based on the code you so kindly provided, for both German and English language documents. See meta.tex.stackexchange.com/q/2884/5001 for more details and a link to the GitHub repository. Comments very welcome! In the package's user guide I list a few lingering bugs, probably caused by particulars of the lua code. I'm entertaining some hope that you may be able to take a look at them and come up with bug fixes. :-) – Mico Nov 21 '12 at 18:09