6

I am using the mechanism outlined in this answer to generate a text-only version of a fairly complex document for spell-checking. This is my first attempt to use lualatex, so this might be littered with issues, but for the most part it does what I want it to do:

\documentclass{article}
\usepackage{luatexbase}
\usepackage{lipsum}
\usepackage{filecontents}
\usepackage{ifluatex}

\begin{filecontents*}{luaFunctions.lua}
-- clear the file
file = io.open("output.txt", "w")
file:write()

exportParagraph = false

function exportText (head)

    if exportParagraph == false then
        --if you return nil no pdf will be created
        -- return nil 
        return head        
    end

    -- open the file in append-modus
    local out = io.open("output.txt", "a")
    local wordCounter = 0

    -- loop over all hboxes in the current paragraph
    for line in node.traverse_id (node.id("hlist"), head) do

        -- loop over each element in the line
        for item in node.traverse (line.list) do
            -- check if the element is a char
            if item.id == node.id("glyph") then
                out:write(string.char(item.char))
            -- check if the element is a 'space'
            elseif item.id == node.id("glue") then
                wordCounter = wordCounter + 1
                out:write(" ")
            end
        end
        -- a newline in the file after each (tex)line
        out:write("\n")        
    end

    wordCounter = wordCounter - 1
    out:write("Words: "..wordCounter.."\n")

    -- a newline in the file after each paragraph 
    out:write("\n")  

    assert(out:close())  
    exportParagraph = false  

    --if you return nil no pdf will be created
    -- return nil
    return head    
end


function disableLigatures(head)
    -- disable ligatures
end

function SetExportParagraph(export)
    exportParagraph = export
end

luatexbase.add_to_callback("ligaturing", disableLigatures, "disableLigatures")
luatexbase.add_to_callback("post_linebreak_filter", exportText, "exportText")
\end{filecontents*}

\ifluatex
    \directlua{dofile("luaFunctions.lua")}
\fi

\def\exportParagraph{%
    \ifluatex
        \directlua{SetExportParagraph(true)}
    \fi
}

\begin{document}
\exportParagraph 
ff fi Lorem ipsum dolor sit amet, \textbf{consectetuer adipiscing elit. Ut purus elit,
vestibulum ut, placerat ac, adipiscing vitae, felis.} Curabitur dictum gravida
mauris. Nam arcu libero, nonummy eget, consectetuer id, vulputate a, magna.
Donec vehicula augue eu neque. Pellentesque habitant morbi tristique senectus
et netus et malesuada fames ac turpis egestas. Mauris ut leo. Cras viverra
metus rhoncus sem. Nulla et lectus vestibulum urna fringilla ultrices. Phasellus
eu tellus sit amet tortor gravida placerat. Integer sapien est, iaculis in, pretium
quis, viverra ac, nunc. Praesent eget sem vel leo ultrices bibendum. Aenean
faucibus. Morbi dolor nulla, malesuada eu, pulvinar at, mollis ac, nulla. Cur-
abitur auctor semper nulla. Donec varius orci eget risus. Duis nibh mi, congue
eu, accumsan eleifend, sagittis quis, diam. Duis eget orci sit amet orci dignissim
rutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrum.

Nam dui ligula, fringilla a, euismod sodales, sollicitudin vel, wisi. Morbi
auctor lorem non justo. Nam lacus libero, pretium at, lobortis vitae, ultricies et,
tellus. Donec aliquet, tortor sed accumsan bibendum, erat ligula aliquet magna,
vitae ornare odio metus a mi. Morbi ac orci et nisl hendrerit mollis. Suspendisse
ut massa. Cras nec ante. Pellentesque a nulla. Cum sociis natoque penatibus et
magnis dis parturient montes, nascetur ridiculus mus. Aliquam tincidunt urna.
Nulla ullamcorper vestibulum turpis. Pellentesque cursus luctus mauris.

\exportParagraph
Nulla malesuada porttitor diam. Donec felis erat, congue non, volutpat at,
tincidunt tristique, libero. Vivamus viverra fermentum felis. Donec nonummy
pellentesque ante. Phasellus adipiscing semper elit. Proin fermentum massa
ac quam. Sed diam turpis, molestie vitae, placerat a, molestie nec, leo. Mae-
cenas lacinia. 

Nam ipsum ligula, eleifend at, accumsan nec, suscipit a, ipsum.
Morbi blandit ligula feugiat magna. Nunc eleifend consequat lorem. Sed lacinia
nulla vitae enim. Pellentesque tincidunt purus vel magna. Integer non enim.
Praesent euismod nunc eu purus. Donec bibendum quam in tellus. Nullam cur-
sus pulvinar lectus. Donec et mi. Nam vulputate metus eu enim. Vestibulum
pellentesque felis eu massa.
\end{document}

In the generated output, the nonsense word rutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrum at the end of the first paragraph will be hyphenated:

[...]
ac, nulla. Cur- abitur auctor semper nulla. Donec varius orci eget risus. Duis 
nibh mi, congue eu, accumsan eleifend, sagittis quis, diam. Duis eget orci sit 
amet orci dignissim rutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrum- 
rutrumrutrumrutrumrutrumrutrumrutrum.  
Words: 134

This happens throughout my entire text and makes spell-checking the text rather hard. Is there a way to disable hyphenation for this hack (I'm hesitant to call it a solution) entirely?

  • this code has more serious issues than hyphenated words. try to load fontspec and select some font, or insert a footnote into counted paragraph... – michal.h21 May 8 '14 at 8:04
  • @michal.h21 I'm open to any suggestions to improve it. I've tried detex with very disappointing results and haven't really found any satisfying out-of-the-box solution to extract the textual content of a document for further processing. – vwegert May 8 '14 at 12:17
  • I think luatex node callbacks are best way to do this. but it is better to use pre_linebreak_filter to do this, because this callback is called before hyphenation and some additional information about node list origin is passed. I am working on a solution – michal.h21 May 8 '14 at 13:13
3

There exists several node processing callbacks in luatex, post_linebreak_filter which you use is not best for your purpose, as you have to deal with node lists broken into lines. Much more suitable is pre_linebreak_filter, which is called before line break.

I also found some bugs in your code, which showed when I tried to use fontspec package and some non ascii characters. First I will post modified file:

\documentclass{article}
\usepackage{luatexbase}
\usepackage{fontspec}
%\setmainfont{TeX Gyre Schola}
\usepackage{lipsum}
\usepackage{filecontents}
\usepackage{ifluatex}

\begin{filecontents*}{luaFunctions.lua}
-- clear the file
local file = io.open("output.txt", "w")
file:write()
file:close()

local char = unicode.utf8.char
exportParagraph = false

function exportText (head, listtype)

  --[[
  -- it is better to solve this using attributes
  if exportParagraph == false then
  --if you return nil no pdf will be created
  -- return nil 
  return head        
  end --]]

  -- open the file in append-modus
  local out = io.open("output.txt", "a")
  local wordCounter = 0
  local charcount = 0
  local function traverse(h)
    local word = false
    for item in node.traverse (h) do
      local skip = node.has_attribute(item, 
      luatexbase.attributes.wordcounton) 
      if skip == 2 then
        -- check if the element is a char
        if item.id == node.id("glyph") then
          if node.is_node(item.components) then
            traverse(item.components)
          else
            out:write(char(item.char))
            charcount = charcount  + 1
            word = true
          end
        elseif 
          item.id == node.id("hlist") 
          or item.id == node.id("vlist")
          or item.id == node.id("insert")
          or item.id == node.id("adjust")
          then
            -- out:write(item.id..","..item.subtype.."[")
            traverse(item.head)
            -- out:write "]"
            -- check if the element is a 'glue'. this means not only space
          elseif item.id == node.id("glue") and item.subtype == 0 then
            -- glue nodes doesn't have to be spaces, count only after word
            if word then
              wordCounter = wordCounter  + 1
              charcount = charcount + 1
            end
            word = false
            out:write(" ")
          end
        end
      end 
      -- if word then wordCounter = wordCounter + 1 end
    end

    -- loop over all hboxes in the current paragraph
    --for line in node.traverse_id (node.id("hlist"), head) do
    -- loop over each element in the line
    traverse(head)
    -- a newline in the file after each (tex)line
    out:write("\n")        
    --end

    -- wordCounter = wordCounter - 1
    out:write("Words: "..wordCounter)
    out:write(", characters: "..charcount)
    out:write(", list type: "..listtype.."\n")

    -- a newline in the file after each paragraph 
    out:write("\n")  

    assert(out:close())  
    --exportParagraph = false  

    --if you return nil no pdf will be created
    -- return nil
    return head    
  end


  function disableLigatures(head)
    -- disable ligatures
  end

  function SetExportParagraph(export)
    exportParagraph = export
  end

  luatexbase.add_to_callback("ligaturing", disableLigatures, "disableLigatures")
  luatexbase.add_to_callback("pre_linebreak_filter", exportText, "exportText")
\end{filecontents*}

\ifluatex
    \newluatexattribute\wordcounton
    \directlua{dofile("luaFunctions.lua")}
\fi

\def\startExportParagraph{%
    \ifluatex
      \wordcounton = 2
        %\directlua{SetExportParagraph(true)}
    \fi
}

\def\stopExportParagraph{%
    \ifluatex
      \wordcounton = 1
    \fi
}


\begin{document}
\startExportParagraph 
\noindent
ff fi Lorem ipsum dolor sit amet, příliš žluťoučký text s diakritikou 
dash\footnote{you should test some options}  -- \hbox{how does that work?} 
\textbf{consectetuer adipiscing elit. Ut purus elit,
vestibulum ut, placerat ac, adipiscing vitae, felis.} Curabitur dictum gravida
mauris. Nam arcu libero, nonummy eget, consectetuer id, vulputate a, magna.
Donec vehicula augue eu neque. Pellentesque habitant morbi tristique senectus
et netus et malesuada fames ac turpis egestas. Mauris ut leo. Cras viverra
metus rhoncus sem. Nulla et lectus vestibulum urna fringilla ultrices. Phasellus
eu tellus sit amet tortor gravida placerat. Integer sapien est, iaculis in, pretium
quis, viverra ac, nunc. Praesent eget sem vel leo ultrices bibendum. Aenean
faucibus. Morbi dolor nulla, malesuada eu, pulvinar at, mollis ac, nulla. Curabitur auctor semper nulla. Donec varius orci eget risus. Duis nibh mi, congue
eu, accumsan eleifend, sagittis quis, diam. Duis eget orci sit amet orci dignissim
rutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrumrutrum.

\begin{tabular}{ll}
        what & about\\
        tables&?
\end{tabular}

\begin{itemize}
        \item you also want to save itemize
        \item items
\end{itemize}

You can \stopExportParagraph stop word countinh in the middle of \startExportParagraph the paragraph.

Nam dui ligula, fringilla a, euismod sodales, sollicitudin vel, wisi. Morbi
auctor lorem non justo. Nam lacus libero, pretium at, lobortis vitae, ultricies et,
tellus. Donec aliquet, tortor sed accumsan bibendum, erat ligula aliquet magna,
vitae ornare odio metus a mi. Morbi ac orci et nisl hendrerit mollis. Suspendisse
ut massa. Cras nec ante. Pellentesque a nulla. Cum sociis natoque penatibus et
magnis dis parturient montes, nascetur ridiculus mus. Aliquam tincidunt urna.
Nulla ullamcorper vestibulum turpis. Pellentesque cursus luctus mauris.

\stopExportParagraph
Nulla malesuada porttitor diam. Donec felis erat, congue non, volutpat at,
tincidunt tristique, libero. Vivamus viverra fermentum felis. Donec nonummy
pellentesque ante. Phasellus adipiscing semper elit. Proin fermentum massa
ac quam. Sed diam turpis, molestie vitae, placerat a, molestie nec, leo. Mae-
cenas lacinia. 

Nam ipsum ligula, eleifend at, accumsan nec, suscipit a, ipsum.
Morbi blandit ligula feugiat magna. Nunc eleifend consequat lorem. Sed lacinia
nulla vitae enim. Pellentesque tincidunt purus vel magna. Integer non enim.
Praesent euismod nunc eu purus. Donec bibendum quam in tellus. Nullam cur-
sus pulvinar lectus. Donec et mi. Nam vulputate metus eu enim. Vestibulum
pellentesque felis eu massa.
\end{document}

There was global variable file used, which interfered with some variable in fontspec. All private variables should be local!. file also weren't closed.

When dealing with unicode characters, we cannot use string.char function, but we must unicode.utf8.char instead.

Then I rewrote node traversing loop as recursive function, because sub lists may occur in a node list and we must process them as well. See traverse function.

I modified the document interface, introducing two macros: startExportParagraph and stopExportParagraph. luatex mechanism of node attributes is used, allowing more flexible switching of counting, even in the middle of paragraphs. I also added character counting.

I added some more test cases:

ff fi Lorem ipsum dolor sit amet, příliš žluťoučký text s diakritikou 
dash\footnote{you should test some options}  -- \hbox{how does that work?} 
\textbf{consectetuer adipiscing elit. Ut purus elit,...

\begin{tabular}{ll}
        what & about\\
        tables&?
\end{tabular}

\begin{itemize}
        \item you also want to save itemize
        \item items
\end{itemize}

You can \stopExportParagraph stop word countinh in the middle of \startExportParagraph the paragraph.

which is saved in the output.txt as:

1you should test some options
Words: 4, characters: 29, list type: insert

ff fi Lorem ipsum dolor sit amet, příliš žluťoučký text s diakritikou dash1 -- how does that work? consectetuer adipiscing elit. 

  what    about    tables    ?
Words: 4, characters: 20, list type:

     • you also want to save itemize
Words: 7, characters: 30, list type:

     • items
Words: 2, characters: 6, list type:

You can the paragraph.
Words: 4, characters: 22, list type:

as you can see, footnotes produce its own paragraph, which shows before the paragraph they appear in. ligatures are breaked int components, so ffi and fi are counted correctly. But this also causes dash to be broken into --. Bullets in itemize environments are counted as words, I have to investigate how to solve this issue. Also character counts are wrong.

5

If you are not looking for a lua solution (which is no doubt possible) you can use the classic Tex version

 \begin{document}\language-1

will turn off hyphenation

  • Setting \language=-1 is the same as setting \language=0: TeXbook, page 455, fourth double dangerous bend. You probably mean \language255 – egreg May 7 '14 at 22:26
  • Actually, LuaTeX seems to behave differently and indeed setting \language=-1 disables hyphenation. I found no mention of this in the LuaTeX manual. – egreg May 7 '14 at 22:34
  • @egreg yes and yes – David Carlisle May 7 '14 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.