# Reviewing hyphenations with LuaTeX

This question led to a new package:
lua-check-hyphen

I wrote a question (Accessing pointers in LuaTeX) hoping it would lead me to a way of reviewing the hyphenations with LuaTeX, and without external programs. I was encouraged to write this question.

The answers to the question “Check all hyphenations within a document” point to external tools.

I’m hoping for a solution which writes the hyphenation to the log file everytime a word is split across two lines, for easy reviewing.

-

## Update:

The package is now also on CTAN and dwells at:
http://www.ctan.org/pkg/lua-check-hyphen

There is a new experimental package for that purpose: lua-check-hyphen which you can download here: https://github.com/pgundlach/lua-check-hyphen (copy the .sty and the .lua file into the current directory). This will list the hyphenated words in a separate file (ending with .uhy) and has the ability to create a whitelist with "good" hyphenations and you can optionally mark unknown hyphenations.

Consider this input file:

\documentclass{article}
\usepackage{fontspec}
\usepackage[mark]{lua-check-hyphen}

\begin{document}

\hsize 1.3in
A wonderful serenity has taken possession of my entire soul, like these sweet
mornings of spring which I enjoy with my whole heart.

\end{document}


which results in

and a file test.uhy with this content:

seren-ity
posses-sion
morn-ings


Now you can add these lines (if you believe that they are valid breakpoints) to a textfile for example called okhyph.txt and add this to the command LuaCheckHyphen. Let's put the second line into that file and use it in the document:

\begin{filecontents}{okhyph.txt}
posses-sion
\end{filecontents}

\documentclass{article}
\usepackage{fontspec}
\usepackage[mark]{lua-check-hyphen}
\LuaCheckHyphen{whitelist=okhyph.txt}

\begin{document}

\hsize 1.3in
A wonderful serenity has taken possession of my entire soul, like these sweet
mornings of spring which I enjoy with my whole heart.

\end{document}


Now lua-check-hyphen does not complain about that word anymore.

And the file test.uhy now contains only:

morn-ings
seren-ity


You can continue until you have a clean document. If you pass the option final to the package, no hyphenation checking is done.

-
If the hyphenated word contains a ligature, the ligature is written to the file, e.g. ﬀ (U+FB00 LATIN SMALL LIGATURE FF). A tt ligature is not written. – Håkon Malmedal Jun 20 '12 at 9:36
@HåkonMalmedal This should be fixed now. – topskip Jun 20 '12 at 9:57
Patrick: this package look great, I wrote about it on my blog, so that more people can be aware of it! – mbork Jun 22 '12 at 10:04
@mbork great! Next version will have a "nofile" option which prevents the software from writing an external file. – topskip Jun 22 '12 at 10:07
@PatrickGundlach: good idea, especially that there are quite a few questions here about TeX cluttering people's disks with strange files;). – mbork Jun 22 '12 at 12:51

Since you're using Lua(La)TeX, you may want to consider employing the nifty showhyphens package. As its name suggests, this package shows all hyphenation points that TeX considers to be valid, whether or not hyphenation actually occurs or not. While this package doesn't perform precisely the task you're trying to accomplish, I believe that its output is actually even more useful than what you're after: Not only does it tell you where hyphenations occur but it also alerts you to (i) potentially invalid hyphenation points and (ii) hyphenation points that TeX might miss entirely.

Taking your three-paragraph question as an example, a simple MWE using this package might look like this:

% !TEX program = lualatex
\documentclass[letterpaper]{article}
\usepackage{fontspec,showhyphens}

\begin{document}
I wrote a question (Accessing pointers in LuaTeX) hoping it would
lead me to a way of reviewing the hyphenations with LuaTeX, and
without external programs. I was encouraged to write this question.

The answers to the question Check all hyphenations within a document''
point to external tools.

I'm hoping for a solution which writes the hyphenation to the log file
everytime a word is split across two lines, for easy reviewing.
\end{document}


and would produce the following output:

Even though this short text contains only one hyphenated word, the little red markers show one incorrect hyphenation point (Lu-a-TeX) and one missed hyphenation point (in the word hyphenation, and in that word's plural form). To fix these issues, it suffices to insert the command

\hyphenation{Lua-TeX hy-phen-a-tion hy-phen-a-tions}


in the preamble of the MWE, leading to the following, more satisfactory output:

Of course, TeX's hyphenation algorithm isn't perfect (nor does anyone claim that it is!), so for "real" production work you should probably want to load a custom hyphenation exception list to treat words that you (or others) have noticed in the past are not hyphenated properly. One such list, aimed at US English documents, is in the file ushyphex.tex that's available on the CTAN; it contains about 1300 words. [Interestingly, the list in this file does not (yet) include treatments of the words LuaTeX and hyphenation(s)!] Of course, the recommendation to load hyphenation exception lists applies to other languages as well, not just to (US) English.

-
+1 for finding that "hyphenation" isn't fully hyphenated. into the exceptions list it goes. – barbara beeton Jun 19 '12 at 20:19
+1 for using showhyphens :-)) – topskip Jun 19 '12 at 20:22
Off-topic, I noticed that your images are always of poor resolution, now I just realised they are poorly scaled by the browser, so may be you would consider generating them at smaller size by default (width of 800px or 600px I think). – Khaled Hosny Jun 19 '12 at 23:23
@KhaledHosny -- Many thanks for this. I'll try to figure out what's going on and (most likely!) change the size per your suggestion. – Mico Jun 20 '12 at 0:04

A slightly more complex answer: This solution collects all possible hyphen points before it breaks the paragraph into lines, after the paragraph is broken into lines it checks again which of these hyphenation have really been used. These are collected in a table and at the end of the document these words are printed to the log file (and to the LuaTeX run).

\documentclass[ngerman]{article}
\usepackage[EU2]{fontenc}
\usepackage{lmodern}
\usepackage{babel}
\usepackage{luacode}
\usepackage{luatexbase-attr}

\AtEndDocument{\directlua{listhyphenatedwords()}}
\begin{document}

\begin{luacode*}

local hyphenattr =  luatexbase.new_attribute("hyphenattr")
do
local hyphenwords = {}
local all_hyphenatedwords = {}

local word_start
local word
-- this is where we store all the breakpoints
local thisbreakpoint
local word_with_hyphen
local c
local hyphencounter = #hyphenwords + 1
local sln = unicode.utf8

elseif head.id == 7 then --disc
while word_start.prev and word_start.prev.id ~= 10 do
word_start = word_start.prev
end
word = ""
c = 0
while word_start and word_start.id ~= 10 do
if word_start == head then -- disc
-- there is a breakpoint after letter c
thisbreakpoint = c
elseif word_start.id == 37 then
c = c + 1
if sln.match(sln.char(word_start.char),"%a") then
word = word .. sln.char(word_start.char)
end
end
word_start = word_start.next
end
word_with_hyphen = sln.sub(word,1,thisbreakpoint) .. "-" .. sln.sub(word,thisbreakpoint+1,-1)
hyphenwords[hyphencounter] = word_with_hyphen
hyphencounter = #hyphenwords + 1
end
end
return true
end

local c
all_hyphenatedwords[#all_hyphenatedwords + 1] = hyphenwords[c]
end
end
return true
end

function listhyphenatedwords()
texio.write_nl("All hyphenated words follow:")
for i in ipairs(all_hyphenatedwords) do
texio.write_nl(all_hyphenatedwords[i])
end
texio.write_nl("All hyphenated words above.")
end
end

\end{luacode*}
\hsize 2in

Seit zwei Jahren ist meine Arbeit in Düsseldorf. Meine Familie lebt dagegen in
Hamburg. Und dazwischen ich, aber ganz cool. Vollbremskombination aus Ampel
oder Einfädeln oder beides auf einmal. Geht nur mit Tricks. Eben noch kurz auf
die A 52. Schon vielversprechend lebhaft. Hinter dem Breitscheider Kreuz geht
es richtig los. Stau auf der A 3 bis Oberhausen. Danach entspannt es sich auch
nur deshalb, weil enge Baustellen mit rüden Geschwindigkeitsbegrenzungen zum
gleichmäßigen Rollen zwingen. Es wird links überholt, es wird rechts überholt.
Es wird gar nichts mehr, alles steht und macht lange Gesichter. Ich dagegen
wechsle die Cassette.

Es geht weiter. Vor und hinter mir hektische Spurenwechsel. So zieht sich das
Stück A 2 bis zum Recklinghauser Kreuz. Der erhofften Entspannung folgt
zuverlässig der Vollfrust. Die A 43 bis Münster ist genauso voll wie das
Kamener Kreuz zur Rush-hour.

\end{document}


gives:

...
All hyphenated words follow:
Ar-beit
Fami-lie
ein-mal
rich-tig
Ober-hausen
Bau-stellen
Geschwindigkeits-begrenzungen
über-holt
hin-ter
zu-verlässig
All hyphenated words above. [1
...


which is in accordance to the output file:

-
The next step would be to have a word list of "correct" hyphenation points and only output / mark in the output the unknown hyphenation points. – topskip Jun 19 '12 at 20:21
Wow, this is a lot more sophisticated than the answer I came up with! It also really shows the power and versatility of lua as a scripting language. – Mico Jun 19 '12 at 21:40