Take the 2-minute tour ×
TeX - LaTeX Stack Exchange is a question and answer site for users of TeX, LaTeX, ConTeXt, and related typesetting systems. It's 100% free, no registration required.

This question led to a new package:
lua-check-hyphen

I wrote a question (Accessing pointers in LuaTeX) hoping it would lead me to a way of reviewing the hyphenations with LuaTeX, and without external programs. I was encouraged to write this question.

The answers to the question “Check all hyphenations within a document” point to external tools.

I’m hoping for a solution which writes the hyphenation to the log file everytime a word is split across two lines, for easy reviewing.

share|improve this question
add comment

3 Answers

up vote 14 down vote accepted

Update:

The package is now also on CTAN and dwells at:
http://www.ctan.org/pkg/lua-check-hyphen


There is a new experimental package for that purpose: lua-check-hyphen which you can download here: https://github.com/pgundlach/lua-check-hyphen (copy the .sty and the .lua file into the current directory). This will list the hyphenated words in a separate file (ending with .uhy) and has the ability to create a whitelist with "good" hyphenations and you can optionally mark unknown hyphenations.

Consider this input file:

\documentclass{article}
\usepackage{fontspec}
\usepackage[mark]{lua-check-hyphen}

\begin{document}

\hsize 1.3in
A wonderful serenity has taken possession of my entire soul, like these sweet
mornings of spring which I enjoy with my whole heart.

\end{document}

which results in

image with boxes

and a file test.uhy with this content:

seren-ity
posses-sion
morn-ings

Now you can add these lines (if you believe that they are valid breakpoints) to a textfile for example called okhyph.txt and add this to the command LuaCheckHyphen. Let's put the second line into that file and use it in the document:

\begin{filecontents}{okhyph.txt}
posses-sion
\end{filecontents}

\documentclass{article}
\usepackage{fontspec}
\usepackage[mark]{lua-check-hyphen}
\LuaCheckHyphen{whitelist=okhyph.txt}

\begin{document}

\hsize 1.3in
A wonderful serenity has taken possession of my entire soul, like these sweet
mornings of spring which I enjoy with my whole heart.

\end{document}

Now lua-check-hyphen does not complain about that word anymore.

enter image description here

And the file test.uhy now contains only:

morn-ings
seren-ity

You can continue until you have a clean document. If you pass the option final to the package, no hyphenation checking is done.

share|improve this answer
    
If the hyphenated word contains a ligature, the ligature is written to the file, e.g. (U+FB00 LATIN SMALL LIGATURE FF). A tt ligature is not written. –  Håkon Malmedal Jun 20 '12 at 9:36
1  
@HåkonMalmedal This should be fixed now. –  topskip Jun 20 '12 at 9:57
1  
Patrick: this package look great, I wrote about it on my blog, so that more people can be aware of it! –  mbork Jun 22 '12 at 10:04
1  
@mbork great! Next version will have a "nofile" option which prevents the software from writing an external file. –  topskip Jun 22 '12 at 10:07
    
@PatrickGundlach: good idea, especially that there are quite a few questions here about TeX cluttering people's disks with strange files;). –  mbork Jun 22 '12 at 12:51
add comment

Since you're using Lua(La)TeX, you may want to consider employing the nifty showhyphens package. As its name suggests, this package shows all hyphenation points that TeX considers to be valid, whether or not hyphenation actually occurs or not. While this package doesn't perform precisely the task you're trying to accomplish, I believe that its output is actually even more useful than what you're after: Not only does it tell you where hyphenations occur but it also alerts you to (i) potentially invalid hyphenation points and (ii) hyphenation points that TeX might miss entirely.

Taking your three-paragraph question as an example, a simple MWE using this package might look like this:

% !TEX program = lualatex
\documentclass[letterpaper]{article}
\usepackage{fontspec,showhyphens}
\addfontfeature{ligatures=TeX}

\begin{document}
I wrote a question (Accessing pointers in LuaTeX) hoping it would 
lead me to a way of reviewing the hyphenations with LuaTeX, and 
without external programs. I was encouraged to write this question.

The answers to the question ``Check all hyphenations within a document'' 
point to external tools.

I'm hoping for a solution which writes the hyphenation to the log file 
everytime a word is split across two lines, for easy reviewing.
\end{document}

and would produce the following output:

enter image description here

Even though this short text contains only one hyphenated word, the little red markers show one incorrect hyphenation point (Lu-a-TeX) and one missed hyphenation point (in the word hyphenation, and in that word's plural form). To fix these issues, it suffices to insert the command

\hyphenation{Lua-TeX hy-phen-a-tion hy-phen-a-tions}

in the preamble of the MWE, leading to the following, more satisfactory output:

enter image description here

Of course, TeX's hyphenation algorithm isn't perfect (nor does anyone claim that it is!), so for "real" production work you should probably want to load a custom hyphenation exception list to treat words that you (or others) have noticed in the past are not hyphenated properly. One such list, aimed at US English documents, is in the file ushyphex.tex that's available on the CTAN; it contains about 1300 words. [Interestingly, the list in this file does not (yet) include treatments of the words LuaTeX and hyphenation(s)!] Of course, the recommendation to load hyphenation exception lists applies to other languages as well, not just to (US) English.

share|improve this answer
5  
+1 for finding that "hyphenation" isn't fully hyphenated. into the exceptions list it goes. –  barbara beeton Jun 19 '12 at 20:19
2  
+1 for using showhyphens :-)) –  topskip Jun 19 '12 at 20:22
3  
Off-topic, I noticed that your images are always of poor resolution, now I just realised they are poorly scaled by the browser, so may be you would consider generating them at smaller size by default (width of 800px or 600px I think). –  Khaled Hosny Jun 19 '12 at 23:23
    
@KhaledHosny -- Many thanks for this. I'll try to figure out what's going on and (most likely!) change the size per your suggestion. –  Mico Jun 20 '12 at 0:04
add comment

A slightly more complex answer: This solution collects all possible hyphen points before it breaks the paragraph into lines, after the paragraph is broken into lines it checks again which of these hyphenation have really been used. These are collected in a table and at the end of the document these words are printed to the log file (and to the LuaTeX run).

\documentclass[ngerman]{article}
\usepackage{luaotfload}
\usepackage[EU2]{fontenc}
\usepackage{lmodern}
\usepackage{babel}
\usepackage{luacode}
\usepackage{luatexbase-attr}

\AtEndDocument{\directlua{listhyphenatedwords()}}
\begin{document}

\begin{luacode*}

local hyphenattr =  luatexbase.new_attribute("hyphenattr")
do
  local hyphenwords = {}
  local all_hyphenatedwords = {}

  function collect_discs (head)
    local word_start
    local word
    -- this is where we store all the breakpoints
    local thisbreakpoint
    local word_with_hyphen
    local c
    local hyphencounter = #hyphenwords + 1
    local sln = unicode.utf8

    while head do
      if head.id == 0 then
      elseif head.id == 7 then --disc
        word_start = head
        word_end   = head
        while word_start.prev and word_start.prev.id ~= 10 do
          word_start = word_start.prev
        end
        word = ""
        c = 0
        while word_start and word_start.id ~= 10 do
          if word_start == head then -- disc
            -- there is a breakpoint after letter c
            node.set_attribute(head,hyphenattr,hyphencounter)
            thisbreakpoint = c
          elseif word_start.id == 37 then
            c = c + 1
            if sln.match(sln.char(word_start.char),"%a") then
              word = word .. sln.char(word_start.char)
            end
          end
          word_start = word_start.next
        end
        word_with_hyphen = sln.sub(word,1,thisbreakpoint) .. "-" .. sln.sub(word,thisbreakpoint+1,-1)
        hyphenwords[hyphencounter] = word_with_hyphen
        hyphencounter = #hyphenwords + 1
      end
      head = head.next
    end
    return true
  end

  function check_discs(head)
    local c
    while head do
      if head.id < 2 then
        check_discs(head.list)
      elseif head.id == 7 and head.next and head.next.id == 10 then -- disc
        c = node.has_attribute(head,hyphenattr)
        all_hyphenatedwords[#all_hyphenatedwords + 1] = hyphenwords[c]
      end
      head = head.next
    end
    return true
  end

  function listhyphenatedwords()
    texio.write_nl("All hyphenated words follow:")
    for i in ipairs(all_hyphenatedwords) do
      texio.write_nl(all_hyphenatedwords[i])
    end
    texio.write_nl("All hyphenated words above.")
  end
end



luatexbase.add_to_callback("pre_linebreak_filter",collect_discs,"collect_discs")
luatexbase.add_to_callback("post_linebreak_filter",check_discs,"check_discs")
\end{luacode*}
\hsize 2in

Seit zwei Jahren ist meine Arbeit in Düsseldorf. Meine Familie lebt dagegen in
Hamburg. Und dazwischen ich, aber ganz cool. Vollbremskombination aus Ampel
oder Einfädeln oder beides auf einmal. Geht nur mit Tricks. Eben noch kurz auf
die A 52. Schon vielversprechend lebhaft. Hinter dem Breitscheider Kreuz geht
es richtig los. Stau auf der A 3 bis Oberhausen. Danach entspannt es sich auch
nur deshalb, weil enge Baustellen mit rüden Geschwindigkeitsbegrenzungen zum
gleichmäßigen Rollen zwingen. Es wird links überholt, es wird rechts überholt.
Es wird gar nichts mehr, alles steht und macht lange Gesichter. Ich dagegen
wechsle die Cassette.


Es geht weiter. Vor und hinter mir hektische Spurenwechsel. So zieht sich das
Stück A 2 bis zum Recklinghauser Kreuz. Der erhofften Entspannung folgt
zuverlässig der Vollfrust. Die A 43 bis Münster ist genauso voll wie das
Kamener Kreuz zur Rush-hour.

\end{document}

gives:

...
All hyphenated words follow:
Ar-beit
Fami-lie
ein-mal
rich-tig
Ober-hausen
Bau-stellen
Geschwindigkeits-begrenzungen
über-holt
hin-ter
zu-verlässig
All hyphenated words above. [1
...

which is in accordance to the output file:

hyphenated text

share|improve this answer
1  
The next step would be to have a word list of "correct" hyphenation points and only output / mark in the output the unknown hyphenation points. –  topskip Jun 19 '12 at 20:21
    
Wow, this is a lot more sophisticated than the answer I came up with! It also really shows the power and versatility of lua as a scripting language. –  Mico Jun 19 '12 at 21:40
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.