Reviewing hyphenations with LuaTeX

Question

This question led to a new package:
lua-check-hyphen

I wrote a question (Accessing pointers in LuaTeX) hoping it would lead me to a way of reviewing the hyphenations with LuaTeX, and without external programs. I was encouraged to write this question.

The answers to the question “Check all hyphenations within a document” point to external tools.

I’m hoping for a solution which writes the hyphenation to the log file everytime a word is split across two lines, for easy reviewing.

Answer 1

Update:

The package is now also on CTAN and dwells at:
http://www.ctan.org/pkg/lua-check-hyphen

---

There is a new experimental package for that purpose: lua-check-hyphen which you can download here: https://github.com/pgundlach/lua-check-hyphen (copy the .sty and the .lua file into the current directory). This will list the hyphenated words in a separate file (ending with .uhy) and has the ability to create a whitelist with "good" hyphenations and you can optionally mark unknown hyphenations.

Consider this input file:

\documentclass{article}
\usepackage{fontspec}
\usepackage[mark]{lua-check-hyphen}

\begin{document}

\hsize 1.3in
A wonderful serenity has taken possession of my entire soul, like these sweet
mornings of spring which I enjoy with my whole heart.

\end{document}

which results in

and a file test.uhy with this content:

seren-ity
posses-sion
morn-ings

Now you can add these lines (if you believe that they are valid breakpoints) to a textfile for example called okhyph.txt and add this to the command LuaCheckHyphen. Let's put the second line into that file and use it in the document:

\begin{filecontents}{okhyph.txt}
posses-sion
\end{filecontents}

\documentclass{article}
\usepackage{fontspec}
\usepackage[mark]{lua-check-hyphen}
\LuaCheckHyphen{whitelist=okhyph.txt}

\begin{document}

\hsize 1.3in
A wonderful serenity has taken possession of my entire soul, like these sweet
mornings of spring which I enjoy with my whole heart.

\end{document}

Now lua-check-hyphen does not complain about that word anymore.

And the file test.uhy now contains only:

morn-ings
seren-ity

You can continue until you have a clean document. If you pass the option final to the package, no hyphenation checking is done.

Answer 2

Since you're using Lua(La)TeX, you may want to consider employing the nifty showhyphens package. As its name suggests, this package shows all hyphenation points that TeX considers to be valid, whether or not hyphenation actually occurs or not. While this package doesn't perform precisely the task you're trying to accomplish, I believe that its output is actually even more useful than what you're after: Not only does it tell you where hyphenations occur but it also alerts you to (i) potentially invalid hyphenation points and (ii) hyphenation points that TeX might miss entirely.

Taking your three-paragraph question as an example, a simple MWE using this package might look like this:

% !TEX program = lualatex
\documentclass[letterpaper]{article}
\usepackage{fontspec,showhyphens}
\addfontfeature{ligatures=TeX}

\begin{document}
I wrote a question (Accessing pointers in LuaTeX) hoping it would 
lead me to a way of reviewing the hyphenations with LuaTeX, and 
without external programs. I was encouraged to write this question.

The answers to the question ``Check all hyphenations within a document'' 
point to external tools.

I'm hoping for a solution which writes the hyphenation to the log file 
everytime a word is split across two lines, for easy reviewing.
\end{document}

and would produce the following output:

Even though this short text contains only one hyphenated word, the little red markers show one incorrect hyphenation point (Lu-a-TeX) and one missed hyphenation point (in the word hyphenation, and in that word's plural form). To fix these issues, it suffices to insert the command

\hyphenation{Lua-TeX hy-phen-a-tion hy-phen-a-tions}

in the preamble of the MWE, leading to the following, more satisfactory output:

Of course, TeX's hyphenation algorithm isn't perfect (nor does anyone claim that it is!), so for "real" production work you should probably want to load a custom hyphenation exception list to treat words that you (or others) have noticed in the past are not hyphenated properly. One such list, aimed at US English documents, is in the file ushyphex.tex that's available on the CTAN; it contains about 1300 words. [Interestingly, the list in this file does not (yet) include treatments of the words LuaTeX and hyphenation(s)!] Of course, the recommendation to load hyphenation exception lists applies to other languages as well, not just to (US) English.

Answer 3

A slightly more complex answer: This solution collects all possible hyphen points before it breaks the paragraph into lines, after the paragraph is broken into lines it checks again which of these hyphenation have really been used. These are collected in a table and at the end of the document these words are printed to the log file (and to the LuaTeX run).

\documentclass[ngerman]{article}
\usepackage{luaotfload}
\usepackage[EU2]{fontenc}
\usepackage{lmodern}
\usepackage{babel}
\usepackage{luacode}
\usepackage{luatexbase-attr}

\AtEndDocument{\directlua{listhyphenatedwords()}}
\begin{document}

\begin{luacode*}

local hyphenattr =  luatexbase.new_attribute("hyphenattr")
do
  local hyphenwords = {}
  local all_hyphenatedwords = {}

  function collect_discs (head)
    local word_start
    local word
    -- this is where we store all the breakpoints
    local thisbreakpoint
    local word_with_hyphen
    local c
    local hyphencounter = #hyphenwords + 1
    local sln = unicode.utf8

    while head do
      if head.id == 0 then
      elseif head.id == 7 then --disc
        word_start = head
        word_end   = head
        while word_start.prev and word_start.prev.id ~= 10 do
          word_start = word_start.prev
        end
        word = ""
        c = 0
        while word_start and word_start.id ~= 10 do
          if word_start == head then -- disc
            -- there is a breakpoint after letter c
            node.set_attribute(head,hyphenattr,hyphencounter)
            thisbreakpoint = c
          elseif word_start.id == 37 then
            c = c + 1
            if sln.match(sln.char(word_start.char),"%a") then
              word = word .. sln.char(word_start.char)
            end
          end
          word_start = word_start.next
        end
        word_with_hyphen = sln.sub(word,1,thisbreakpoint) .. "-" .. sln.sub(word,thisbreakpoint+1,-1)
        hyphenwords[hyphencounter] = word_with_hyphen
        hyphencounter = #hyphenwords + 1
      end
      head = head.next
    end
    return true
  end

  function check_discs(head)
    local c
    while head do
      if head.id < 2 then
        check_discs(head.list)
      elseif head.id == 7 and head.next and head.next.id == 10 then -- disc
        c = node.has_attribute(head,hyphenattr)
        all_hyphenatedwords[#all_hyphenatedwords + 1] = hyphenwords[c]
      end
      head = head.next
    end
    return true
  end

  function listhyphenatedwords()
    texio.write_nl("All hyphenated words follow:")
    for i in ipairs(all_hyphenatedwords) do
      texio.write_nl(all_hyphenatedwords[i])
    end
    texio.write_nl("All hyphenated words above.")
  end
end



luatexbase.add_to_callback("pre_linebreak_filter",collect_discs,"collect_discs")
luatexbase.add_to_callback("post_linebreak_filter",check_discs,"check_discs")
\end{luacode*}
\hsize 2in

Seit zwei Jahren ist meine Arbeit in Düsseldorf. Meine Familie lebt dagegen in
Hamburg. Und dazwischen ich, aber ganz cool. Vollbremskombination aus Ampel
oder Einfädeln oder beides auf einmal. Geht nur mit Tricks. Eben noch kurz auf
die A 52. Schon vielversprechend lebhaft. Hinter dem Breitscheider Kreuz geht
es richtig los. Stau auf der A 3 bis Oberhausen. Danach entspannt es sich auch
nur deshalb, weil enge Baustellen mit rüden Geschwindigkeitsbegrenzungen zum
gleichmäßigen Rollen zwingen. Es wird links überholt, es wird rechts überholt.
Es wird gar nichts mehr, alles steht und macht lange Gesichter. Ich dagegen
wechsle die Cassette.


Es geht weiter. Vor und hinter mir hektische Spurenwechsel. So zieht sich das
Stück A 2 bis zum Recklinghauser Kreuz. Der erhofften Entspannung folgt
zuverlässig der Vollfrust. Die A 43 bis Münster ist genauso voll wie das
Kamener Kreuz zur Rush-hour.

\end{document}

gives:

...
All hyphenated words follow:
Ar-beit
Fami-lie
ein-mal
rich-tig
Ober-hausen
Bau-stellen
Geschwindigkeits-begrenzungen
über-holt
hin-ter
zu-verlässig
All hyphenated words above. [1
...

which is in accordance to the output file: