highlighting words which cannot be hyphenated

Question

is there any package which can automatically highlight unhyphenable words in output PDF? I.e. words which are not in the list of words for hyphenation. For example:

I'm using polyglossia package. Thank you very much!

Answer 1

A proof of concept

We've checked some common packages, such as ulem (Don­ald Arse­neau) and soul (Mel­chior Franz), after that we've moved our attention to the showhyphens, lua-check-hyphen and lua-visual-debug packages created by Patrick Gundlach. Unfortunately, the OP needs the opposite from what these packages offer.

We started our work by analysing and learning from this code from TeX.SX (see also http://wiki.luatex.org/index.php/Show_the_hyphenation_points) by Patrick Gundlach. We are always supported by the reference manual of LuaTeX, where we've noticed defined types of nodes (see Chapter 8 Nodes, first page; page 169+ in the manual). We can read about their fields on the following pages.

We especially needed these types:

• hlist (0), horizontal lists,
• vlist (1), vertical lists,
• disc (7), a place where a word could be hyphenated,
• whasit (8), suitable type for a new node creation,
• glue (10), spaces,
• kern (11), kernings, and,
• glyph (37), the key node type for this sort of work.

Our task was to go through all nodes, collect and store their ID in a Lua table, do some basic analysis at a text level and after that do some basic highlighting of the results (in this example, we used underlining of every glyph).

We are passing a minimum number of letters which must have that word to be highlighted. A single word consists of letters, digits, punctuations etc. We believe that should be sufficient for this proof of concept.

The original code to show hyphens uses recursive algorithm to go through the node tree. We needed two passes, therefore we wrapped that function (show_hyph) to another one (towork). The analysis is done in the showme function. We use head userdata, we could print the fields using the getmetatable function. It is commented out in the code as well as printing additional information about the nodes.

Text analysis consists of several conditions, mainly from:

• ID 0 or ID 10 followed by ID 37 (plus the word cannot be already hyphenated), that's a beginning of a word. It is a new line or a space followed by a glyph.
• ID 7 means that there can be a disc and that word is of no interest to us.
• ID 11 (kerning) or ID 37 (glyph) is a part of a word, we include that part for further analysis.
• ID 37 followed by ID 10 is the end of a word.

We store all information in a Lua table. We needed two variables about a length of the word, for displaying a word we used malsizetrue (glyphs+kerning nodes), for interaction with a user we used malsize (glyphs only). It was strange at first to think about words in these terms, but it was necessary.

We used head.width (sp units) to get proper underlining. In the example below, every letter is underlined separately, it could be improved to draw one single line per word. We are leaving this idea opened for further experiments with the code. That would require to store widths in the first run of the function and using them in the second run at a Lua level.

In the following example, we used some basic text and we set a minimum number of letters per word from 1 to 8. We enclose the code and a preview of the result (8 pages). It works as expected in this example, but I guess that this snippet requires more testing.

We run lualatex mal-hyph.tex.

% lualatex mal-hyph.tex
\documentclass[a4paper]{article}
\pagestyle{empty}
\usepackage{luacode} % the core package in LuaLaTeX, recommended is luatextra
\usepackage[english]{babel} % loading some hyphenation rules
\usepackage{kantlipsum} % loading some text
\usepackage{fontspec} % loading some font

\begin{luacode*}
-- This is the core function for callback...
function towork(head)

-- This inner function is recursive...
function show_hyph(head)
while head do

  -- If we want to see userdata of >head<...
  -- for i,j in pairs(getmetatable(head)) do
  --    print(i,j)
  -- end

  -- If we need to get nodes ID..., this is the way.
  --[=[  io.write(head.id.." ")
    if head.id ==10 then io.write(" ") end
    if head.id == 37 then 
       io.write(unicode.utf8.char(head.char).."; ")
       end    
    --]=] -- be silent

    if head.id == 0 or head.id == 1 then
       show_hyph(head.head) -- the core of recursion
    end -- if, head.id, 0 (hlist) or 1 (vlist)

    counter=counter+1 -- number of nodes
    malglyphs[counter]=head.id -- saving IDs for later analysis

if run then -- We can add nodes after analysis...
  if maldraw[counter]==1 then -- These letters belong to unhyphenateable words.
      local n = node.new("whatsit","pdf_literal") -- Underline them, please,
      n.mode = 0
      if head.width then dista=head.width/65535 else dista=0 end -- with proper  width,
      distb=0
      n.data = "q 1 w 0 0 1 RG "..-dista.." -2 m "..distb.." -2 l S Q" -- with some common line.
      n.next = head.next -- Add line to the node tree.
      n.prev = head
      head.next = n
      head = n
  end -- if we are done with lines.

    if head.id == 7 and run then -- a divis, this is an original example, addition of disc
      local n = node.new("whatsit","pdf_literal")
      n.mode = 0
      n.data = "q 0.9 w 1 .1 .1 RG 0 2 m 0 7 l S Q" -- an original setting was: q 0.3 w 0 2 m 0 7 l S Q
      n.next = head.next
      n.prev = head
      head.next = n
      head = n
    end -- if, head.id==7
end -- if, run

head = head.next -- go to the next node
end -- while, node three

end -- function, show_hyph


-- This function analyses ID of nodes.
function showme()
-- Initialize some variables.
malwrite=0 -- Can I store a unhyphenateable word?
maldraw={} -- Table with 1s for storing information.
malsize=0 -- A size of the last word (only letters, ID 37).
malsizetrue=0 -- A size of the last word (letters+kerning, IDs 11+37).
broken=0 -- Is a word already hyphenated?

for malcounter=1,#malglyphs do -- Let's analyse all nodes.

-- Print some letters from actual position backward.
-- print(malglyphs[malcounter-2], malglyphs[malcounter-1], malglyphs[malcounter]) 

-- Are we at the beginning of the word?
if  (malglyphs[malcounter-1]==0 or malglyphs[malcounter-1]==10) and malglyphs[malcounter]==37 and broken~=1 then malwrite=1; malsize=0; malsizetrue=0 end

-- Are we at the end of the word? Can we modify data in our Lua table?
if malglyphs[malcounter-1]==37 and (malglyphs[malcounter]==10 or malglyphs[malcounter]==12) then 
   if malwrite==1 and malsize>=malmax then -- used to be: malsize>0
   for malback=1,malsizetrue do -- set letters of unhyphenated word for underlining/highlighting, plus kerning
      maldraw[malcounter-malback]=1 -- mark them letter by letter
   end -- for, malback
   end -- if, malwrite
end -- if, malglyphs...

-- Are we in the middle of the hyphenated word?
if malglyphs[malcounter]==7 then broken=1 end -- Yes, we are!
if malglyphs[malcounter]==37 then broken=0 end -- No, at least not at the beginning of the line!

-- Is this a node with letter or kerning in some word?
if malglyphs[malcounter]==37 or malglyphs[malcounter]==11 then 
   malsizetrue=malsizetrue+1 -- letters+kernings
   if malglyphs[malcounter]==37 then malsize=malsize+1 end -- Only letters.
   else 
   malwrite=0 -- Stop analysing actual word, it has no meaning anymore.
   end -- stop searching

end -- for, malcounter

-- If we need some information from the Lua table...
--[[for allchars=1,#malglyphs do
   if maldraw then print(allchars, maldraw[allchars]) end
   end -- for, malheader
   ]]
end -- function, showme


-- This is the first run, we need data about nodes.
malglyphs={}; run=nil; counter=0
show_hyph(head)
showme()

-- This is the second run, we will highlight letters (and also kerning with an invisible pen :-) ).
malglyphs={}; run=1; counter=0
show_hyph(head)
showme()

return true -- Let's finish our work and send data back to a callback.
end -- major function, towork()
\end{luacode*}


% Some TeX commands for general use...
% A minimum of letters required in an unhyphenateable word to be highlighted.
\def\malactivateset#1{
   \luaexec{
   print("malmax "..#1)
   malmax=tonumber(#1) or 0
   }}

% This command activates a callback.
\def\malactivate{
   \directlua{luatexbase.add_to_callback("post_linebreak_filter",towork,"towork")}
   } % End of \activateme...

% This command deactivates a callback.
\def\maldeactivate{
   \directlua{luatexbase.remove_from_callback("post_linebreak_filter","towork")}
   } % End of \deactivateme...

% This text is an example for TeX.SX.
\def\maltext{\newpage
   Minimum of required letters: \the\malletters.\par
   Hello, the World!\par
   Here is some text without meaning.\par 
   \kant[1]\par
   } % End of \maltext...

% Small changes in paper mirror...
\rightskip=7cm
\parindent=0pt


\begin{document}
% A small example...
\malactivate % This command activates the algorithm...
\newcount\malletters 
\malletters=0 % Start with 0+1 letters, a common minimum.
\loop
   \advance\malletters by 1 % Try more letters.
   \malactivateset{\the\malletters} % Pass No. of letters to Lua.
   \maltext \relax % Text to be tested.
\ifnum\malletters<8 \repeat % Show us 8 pages.
\maldeactivate % This command deactivates the algorithm...
\end{document}