# How to suppress the operation of a luatex-defined macro on a string if the string is part of macro or a label

I'm working towards creating a lualatex-based package that lets users automatically suppress the use of ligatures (for now, ff, fi, fl, ffi, ffl, and ft) for selected words. (For background see this question.) The package is set to work with both English and German language words. The MWE below, which is a very much stripped down version of the package, shows how to suppress the insertion of ligatures for four selected words -- two English, two German. (The correct hyphenation of the selected words -- both at the non-ligation points and potentially elsewhere in the words -- is also taken care of. The little red dashes in the output image, generated by of the showhyphens package, indicate where LuaLaTeX thinks it's OK to insert hyphenation breaks.)

Here's the problem I'm trying to solve: The package's main routine (implemented as a lua callback function that operates on process_input_buffer) turns out to be way too greedy for its own good: It tries to perform string substitution operations on everything in the input buffer, including the names and arguments of TeX macros. To make the package suitable for field work, I have to find a way to prevent the main text translation macro from operating on

• string snippets that are parts of TeX macros and on
• the arguments of select instructions, such as \label and \ref.

(There are probably other cases where the substitution shouldn't be applied either.)

Are there any conditionals -- or how might one go about creating such conditionals? -- to check if a string for which a match is found is part of either an already-defined macro or an argument of a \label or \ref (or \varioref, \cref, etc) macro? Alternatively, how might one prevent outright the substitution macro from operating on (i) any TeX macros and (ii) the arguments of selected macros?

A couple of quick illustrations of these problems:

• Suppose that there's a macro in a document named named \bookshelfful. (Not exactly likely, of course, but this is just meant to provide an example.) For such a macro, I don't want my macro operating on it, as it would end up being transformed into \bookshelf \nobreak\hskip0pt \discretionary{\char\hyphenchar\font}{}{\kern\KERN} \nobreak\hskip0pt ful. Arggh.

• Should there be a label in the document named "thm:cufflinks" (yeah, sure!), it must not get translated into "thm:cuff\nobreak\hskip0pt \discretionary{\char\hyphenchar\font}{}{\kern\KERN} \nobreak\hskip0pt links".

Double arrgh.

% !TEX TS-program = lualatex
\documentclass[12pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage[no-math]{fontspec}
%% work around a bug in luaotfload (cf. http://tex.stackexchange.com/q/47031/5001)
\setmainfont[Renderer=Basic]{Latin Modern Roman}
\defaultfontfeatures{Ligatures={TeX,Common}}
\usepackage{showhyphens} % show all hyphenation points
\usepackage{luatexbase,luacode}

\begin{luacode*}

do
local replace = {}

local filter = function ( buf )
for key, val in pairs ( replace ) do
buf = string.gsub ( buf, key, val )
end
return buf
end

function translateinput ( arg1,arg2 )  -- with discretionary hyphen
replace[arg1]=string.gsub(arg2,"|%*|",[[\kernandhyph ]])
end

function enablefilters()
end

end

\end{luacode*}

\newcommand\enableinputtranslation{
\directlua{
enablefilters()
}
}

\newcommand{\kernandhyph}{%
\nobreak\hskip0pt%
\discretionary{\char\hyphenchar\font}{}{\kern\KERN}%
\nobreak\hskip0pt%
}

\newcommand\translateinput[2]{
\directlua{
translateinput ( "\luatexluaescapestring{#1}",
"\luatexluaescapestring{#2}" )
}
}

% some substitution rules
\translateinput{lfful}{lf|*|ful}    %% e.g., shelf-ful(s) bookshelf-ful(s)
\translateinput{iflich}{if|*|lich}  %%       reif-lich begreif-lich tarif-lich
\translateinput{uflauf}{uf|*|lauf}  %%       auf-laufen
\translateinput{ufform}{uf|*|form}  %%       auf-formen

\newlength\KERN
\setlength\KERN{0.07ex}  % trial value for amount of kern to be inserted

\begin{document}

\enableinputtranslation  % turn on input translation

\end{document}

-
I don't think you can catch all commands like \ref, whose contents should be left intact. I think that this problem ought to be solved during the typesetting phase and not the parsing/interpreting phase of the LaTeX run. –  tohecz Mar 18 '12 at 18:59
I agree with @tohecz - a more advanced pre_linebreak_filter should be used. Perhaps one day I'll post another solution to the original question... –  topskip Mar 18 '12 at 20:31
@PatrickGundlach -- Yes please. :-) –  Mico Mar 18 '12 at 21:13
+1 Auflaufform is evil, ligaturewise. –  doncherry Mar 19 '12 at 23:14
@doncherry -- I confess I was quite pleased when I figured out how to make all of the German non-ligature cases (from the rmligs web collection) work properly whether or not the words start with an uppercase or lowercase lower. If and when I (or, much more likely, Patrick Grundlach!) can figure out how to prevent the ligature substitutions from being applied to (i) TeX macros and (ii) the arguments of certain macros such as \label and \ref, I'll be ready to release the package in beta form to the CTAN -- an event that will, no doubt, prompt much more debugging... –  Mico Mar 19 '12 at 23:33

Here is my solution to this problem, which also uses the ligaturing callback (reusing lots of code from the earlier answer).

Instead of attempting to do the actual hyphenation in the processing function, my code one inserts whatsit nodes at the key spots. Those whatsit nodes then prohibit ligature building at those spots.

\documentclass{article}
\usepackage{fontspec}
\usepackage{luacode}%,luatexbase}
\setmainfont[Renderer=Basic]{Latin Modern Roman}
%\defaultfontfeatures{Ligatures={TeX,NoCommon}}
%\setmainfont{Linux Libertine O}
\usepackage[margin=1cm]{geometry}
\begin{luacode}
local glyph = node.id('glyph')
local glue = node.id("glue")
local whatsit = node.id("whatsit")
local userdefined
for n,v in pairs(node.whatsits()) do
if v == 'user_defined' then userdefined = n end
end
local identifier = 123456  -- any unique identifier
local noliga={}
debug=false
function debug_info(s)
if debug then
texio.write_nl(s)
end
end
local blocknode = node.new(whatsit, userdefined)
blocknode.type = 100
blocknode.user_id = identifier

function process_ligatures(nodes,tail)
local s={}
local current_node=nodes--node.copy(nodes)
local build_liga_table =  function(strlen,t)
local p={}
for i = 1, strlen do
p[i]=0
end
for k,v in pairs(t) do
debug_info("Match: "..v[3])
local c= string.find(noliga[v[3]],"|")
local correction=1
while c~=nil do
debug_info("Position "..(v[1]+c))
p[v[1]+c-correction] = 1
c = string.find(noliga[v[3]],"|",c+1)
correction=correction+1
end
end
debug_info("Liga table: "..table.concat(p, ""))
return p
end
local i=1
if ligatures[i]==1 then
debug_info("Current glyph: "..unicode.utf8.char(curr.char))
node.insert_before(hh,curr, node.copy(blocknode))
hh=curr
end
last=curr
if i==#ligatures then
debug_info("Leave node list on position: "..i)
break
end
i=i+1
end
if(last~=nil) then
debug_info("Last char: "..unicode.utf8.char(last.char))
end--]]
end
for t in node.traverse(nodes) do
if t.id==glyph then
s[#s+1]=string.lower(unicode.utf8.char(t.char))
elseif t.id== glue then
local f=string.gsub(table.concat(s,""),"[\\?!,\\.]+","") -- add all interpunction
local throwliga={}
for k, v in pairs(noliga) do
local count=1
local match= string.find(f,k)
while match do
count=match
debug_info("pattern match: "..f .." - "..k)
local n = match + string.len(k)-1
table.insert(throwliga,{match,n,k})
match= string.find(f,k,count+1)
end
end
if #throwliga==0 then
debug_info("No ligature substitution for: "..f)
else
debug_info("Do ligature substitution for: "..f)
local ligabreaks=build_liga_table(f:len(),throwliga)
apply_ligatures(current_node,ligabreaks)
end
s={}
current_node=t
end
end
-- node.ligaturing(nodes) -- not needed, luaotfload does ligaturing
end
function suppress_liga(s,t)
noliga[s]=t
end
function drop_special_nodes (nodes,tail)
for t in node.traverse(nodes) do
if t.id == whatsit and t.subtype == userdefined and t.user_id == identifier then
node.remove(nodes,t)
node.free(t)
end
end
end
\end{luacode}
\newcommand\suppressligature[2]{
\directlua{
suppress_liga("\luatexluaescapestring{#1}","\luatexluaescapestring{#2}")
}
}
\newcommand\debugon{%
\directlua{
debug=true
}
}
\begin{document}

\suppressligature{fifi}{f|ifi}
\suppressligature{grafi}{graf|i}
\suppressligature{lfful}{lf|ful}
\suppressligature{iflich}{if|lich}
\suppressligature{uflauf}{uf|lauf}
\suppressligature{ufform}{uf|form}
\debugon

shelfful
unbegreiflich
Auflaufform
offen

\end{document}

As you can see, the code does not do any ligaturing at all (!) as that is handled by luaotfload in the pre_linebreak_filter.

However, this also creates a minor glitch: the added whatsits also prevent kerning at those spots, but they cannot be removed here because that would re-enable the ligatures once luaotfload comes into play. I do not know enough of the internals of lualatex to fix this (minor) problem.

-
Thanks so much for this brand-new solution. A quick note: Hyphenation seems to be a bit messed up now. E.g., the words "unbegreiflich" and "Auflaufform" would get hyphenated as "un-be-grei-flich" (instead of un-be-greif-lich) and "Au-flauf-form (instead of Auf-lauf-form). Moreover, the newly added word "baffle" now contains a hyphenation point between "ffl" and "e" -- yikes! I haven't studied your code in detail yet, but do you think that inserting suitable \discretionary{\char\hyphenchar\font}{}{} instructions at the no-ligation points is feasible -- or will this re-enable the ligatures? –  Mico Jul 10 '12 at 12:48
@Mico: While adding discretionaries would probably work, the real problem is that those special nodes have to be removed somehow. Perhaps it is possible to add drop_special_nodes to the end of pre_linebreak_filter ? –  Taco Hoekwater Jul 10 '12 at 13:36
You're absolutely right. On further reflection, I've decided it may be wise to keep the chores of ligature suppression and hyphenation separate. And, it's not that much extra work, actually, to set up explicit \hyphenation{...} lists to deal with the words in question. –  Mico Jul 10 '12 at 13:53
I've been working with your MWE and have come across a problem that appears to be related to words that contain an Umlaut before the ligature suppression point. The question is too long to post as a simple follow-up, so I've posted it as a new question -- see tex.stackexchange.com/q/63005/5001. Comments and insights most welcome! –  Mico Jul 11 '12 at 20:34

I just saw this question due to a follow-up question. One other possibilty to disable ligatures in special cases is to change the open type features. As an example with the following testsubs.fea you tell the open type engine to ignore some substitutions (don't ask me about all the square brackets, it was a lot trial and error until I found a syntax that works):

# Script and language coverage
#         languagesystem DFLT dflt;
#         languagesystem latn dflt;

languagesystem DFLT dflt;
languagesystem latn dflt;

feature nolg {
ignore substitute  f' l' [a] [u] [f];
ignore substitute  [i] f' l' [i] [c] [h];
substitute  f' l' by f_l ;
ignore substitute  f' f' [o] [r] [m];
substitute  f' f' by f_f ;
} nolg;

This fea-file can then be used like this:

\documentclass{article}
\usepackage{fontspec}
\setmainfont[FeatureFile=testsubs.fea, RawFeature=+nolg,Ligatures=NoCommon]
{Latin Modern Roman}
\begin{document}
Auflage Pflicht

unbegreiflich Auflaufform
\end{document}

This gives this:

Remark: The standard ligatures must be disabled and declared anew in the fea (I have no idea if and how the "ignore" statements can be added to the normal liga declarations).

A current luotfload will recreate the cache for the font if the feature file has changed so in theory it is possible to create them on-the-fly.

-
Thanks, this is a great contribution! I was completely unaware of this method for affecting the use (or suppression) of ligatures. –  Mico Jul 12 '12 at 14:34

(too long for a comment)

I have tried and tried, but I can't find a good solution. @michal.h21 has tried with the ligaturing callback, but this, as he writes, fails when using complex rendering mode in LuaTeX. So I've tried to hook into the pre_linebreak_filter which has all ligatures in their place, but I have problems breaking the ligatures apart. Say for example you have a ligature ffi and want to suppress ligaturing between f and fi. TeX (or the font subsystem) has already created the ligature when I am in the pre_linebreak_filter and you can break them apart, but only in a predefined way: in the word "fluffiest" the second ligature is made of the ligature "ff" and "i" thus it is "ffi". The ligature "ff" is, obviously, made of "f" and "f". Now it would be easy to split at "ff|i" - I keep the ff-ligature and insert a space before the "i". But to separate the "ffi" at "f|fi" I would have to create a new ligature ("fi"), but this is almost impossible (not technically, I could create the necessary data structures, but the logic how to create ligatures is in the font subsystem). Therefore the pre_linebreak_filter is not a good place to do the un-ligaturing.

But if we move forward (towards the original input of TeX), it is impossible to parse the input and get all occurrences of the words:

\documentclass{article}
\begin{document}
\newcommand\auflauf{Auflauf}
\newcommand\form{form}
\auflauf\form
\end{document}

The word is still "ligatured" but you won't be able to parse this with the method you want in your question.

This leads to my result: it is not possible without hooking into the font subsystem (in this case luaotfload).

-
Thanks, Patrick! Real quick: Might the ability of the \nameref macro to skip over the arguments of \label commands -- see also the recent question "How to use nameref to refer to a section without including the output of any commands in the title of that section?" (tex.stackexchange.com/q/49332/5001) -- be usable as a template for implementing something similar for translateinput? –  Mico Mar 26 '12 at 3:25

There is ligaturing callback in luatex, which should be possible to use be used exactly for this purpose. Edit There is new code and some explanation

\documentclass{article}
%\usepackage{fontspec}
\usepackage{luacode}%,luatexbase}
%\setmainfont[Renderer=Basic]{Latin Modern Roman}
%\defaultfontfeatures{Ligatures={TeX,NoCommon}}
%\setmainfont{Linux Libertine O}
\usepackage[margin=1cm]{geometry}
\begin{luacode}
local glyph = node.id('glyph')
local glue= node.id("glue")
local noliga={}
debug=false
function debug_info(s)
if debug then
texio.write_nl(s)
end
end
function process_ligatures(nodes,tail)
local s={}
local current_node=nodes--node.copy(nodes)
local build_liga_table =  function(strlen,t)
local p={}
for i = 1, strlen do
p[i]=0
end
for k,v in pairs(t) do
debug_info("Match: "..v[3])
local c= string.find(noliga[v[3]],"|")
local correction=1
while c~=nil do
debug_info("Position "..(v[1]+c))
p[v[1]+c-correction] = 1
c = string.find(noliga[v[3]],"|",c+1)
correction=correction+1
end
end
debug_info("Liga table: "..table.concat(p, ""))
return p
end
local i=1
if ligatures[i]==1 then
node.ligaturing(hh,node.prev(curr))
debug_info("Current glyph: "..unicode.utf8.char(curr.char))
hh=curr--node.next(curr)
end
last=curr
if i==#ligatures then
debug_info("Leave node list on position: "..i)
break
end
i=i+1
end
if(last~=nil) then
debug_info("Last char: "..unicode.utf8.char(last.char))
node.ligaturing(hh,last)
end--]]
end
for t in node.traverse(nodes) do
if t.id==glyph then
s[#s+1]=string.lower(unicode.utf8.char(t.char))
elseif t.id== glue then
local f=string.gsub(table.concat(s,""),"[\\?!,\\.]+","") -- add all interpunction
local throwliga={}
for k, v in pairs(noliga) do
local count=1
local match= string.find(f,k)
while match do
count=match
debug_info("pattern match: "..f .." - "..k)
local n = match + string.len(k)-1
table.insert(throwliga,{match,n,k})
match= string.find(f,k,count+1)
end
end
if #throwliga==0 then
debug_info("No ligature substitution for: "..f)
node.ligaturing(current_node,t)
else
local ligabreaks=build_liga_table(f:len(),throwliga)
apply_ligatures(current_node,ligabreaks)
end
--node.ligaturing(current_node,t)
s={}
current_node=t
end
end
end
function suppress_liga(s,t)
noliga[s]=t
end
\end{luacode}
\newcommand\suppressligature[2]{
\directlua{
suppress_liga("\luatexluaescapestring{#1}","\luatexluaescapestring{#2}")
}
}
\newcommand\debugon{%
\directlua{
debug=true
}
}
\begin{document}

\suppressligature{fifi}{f|ifi}
\suppressligature{grafi}{graf|i}
\suppressligature{lfful}{lf|ful}
\suppressligature{iflich}{if|lich}
\suppressligature{uflauf}{uf|lauf}
\suppressligature{ufform}{uf|form}
\debugon

shelfful
unbegreiflich Auflaufform

\end{document}

General idea is following:

Your current solution uses process_input_buffer callback, which enables easy string replacing, but the downside is that these strings contains raw TeX code and it is easy to ruin all the things. I think the right way is to disable ligatures after all TeX macros were applied and node list have been build. There is ligaturing callback, which is used to ligaturing the document.

In the code, we traverse the node list callback had received and filter all glyph nodes (hese contains text), when there is glue node, we can save current word and apply the ligaturing filter.

If the word didn't match any pattern, we can do ligaturing with

node.ligaturing(word_start_node,word_stop_node)

otherwise, we build table, where all points for disabling ligatures are marked with 1, for example:

auflaufform
00010001000

Then we loop through word's nodes and break ligaturing on every occurence of 1.

Result:

This code has two issues, one is bug I cannot find, other one is more serious.

The bug is, if word contains ligature just before ligature break, as in cufflink, all nodes after this point are discarded. I can't find the source of this bug, but I hope this can be solved.

The second issue is more serious, when using fontspec, no filtering is applied, fontspec handles ligatures on its own. And I don't know, if it is possible to suppress fontspec's ligaturing mechanism, so at the moment, in the real world, this code is useless even if we solve the other bug.

-
This looks promising! I guess that, in its present form, something of a shortcoming of the code is that it can't process the case of a group of three characters (e.g., ffi and ffl) which should not be broken up as f-f-i and f-f-l. In such cases, luatex would have to be instructed as to where the ligature should be broken up: e.g., should a potential ffi triple be broken up as f-fi (wolffish and auffischen) or as ff-i (Pfaffian, Wolffian, and Rohstoffindustrie)? –  Mico Mar 22 '12 at 0:26
I have edited my code, now it supports more advanced patterns, but because of issue with fontspec, maybe it is useless –  michal.h21 Mar 23 '12 at 13:37
Your problem is not with fontspec but with open type fonts and luaotfload generally. Ligatures are enabled with the +liga tag and then your code doesn't work. –  Ulrike Fischer Jul 12 '12 at 10:40