LuaLatex: How get ascii text from paragraph, change it and render again

Question

Is possible to render paragraph in LuaTeX -> get ascii text with hyphenation -> put back text and render again?

1. I have simple text: "Lorem ipsum dolor sit amet, conse... laborum."

I need get this text to render in 5 cm width in 12 pt specific font.

1. render text and get array (with some data) Array

   temptext[0]="Lorem ipsum dolor sit a-"
   
   temptext[1]="met, consectetur adipisicing e-"
   
   temptext[2]="lit, sed do eiusmod tempor in-"
   
   ...
   
   temptext[n]="lit anim id est laborum."
   

2. I change some parameters or letter for example new width is 5.1cm not 5.0cm, change some letter , change font etc.

3. while my text have different lines than for example n=21 I change some parameters and go to 2.

---

This algorithm is possible to more than one file. for example my paragraph is \include file.tex and 3. point is run perl script, ruby, exe or bash script and run again. I need only number of lines , array table (what is first letter in line etc), and where are hyphenation mark.

Polish [Mogę napisać po polsku. Chodzi mi o to by dowiedziec się gdzie TeX wstawia znaki podziału wyrazu i tak dobrac parametry by wszystko zmieściło się w zadanym prostokącie (np. jakis skomplikowany tekst ze wzorem), zmieniając np. wielkosc fontu, glify, sam font itp.

Potrzebuje zatem informacji jak bardzo tekst nie pasuje do prostokąta. Gdzie ma błędy (overfull box), czyli gdzie wystaje, gdzie nie wystaje i o ile czyli end space size. Mam kilka glifów reprezentujących te same znaki. np. we wzorach moge użyć kilku fontów, które sa podobne do siebie, albo ligatury moge dorobić swoje byle wszystko pasowało do prostokatu akapitu. Alu musze wiedzieć jak bardzo nie pasuje to do akapitu. Tak by napisac program poprawiający tekst na tyle by błąd dopasowania był jak najmniejszy]

Jeśli ktoś może niech przetłumaczy polski tekst, który może jest jasniejszy na angielski. Dziękuję. ]

Answer 1

A proof of concept:

\documentclass{article}
\usepackage{luacode}
\usepackage{libertine}
\begin{document}
\begin{luacode*}
function count_lines (head)
  local linecount = 0
  while head do
    if head.id == 0 then linecount = linecount + 1 end
    head = head.next
  end
  return linecount
end

function mknodes( text )
  local current_font = font.current()
  local font_parameters = font.getfont(current_font).parameters
  local n, head, last
  -- we should insert the paragraph indentation at the beginning
  head = node.new("glue")
  head.spec = node.new("glue_spec")
  head.spec.width = 20 * 2^16
  last = head

  for s in string.utfvalues( text ) do
    local char = unicode.utf8.char(s)
    if unicode.utf8.match(char,"%s") then
      -- its a space
      n = node.new("glue")
      n.spec = node.new("glue_spec")
      n.spec.width   = font_parameters.space
      n.spec.shrink  = font_parameters.space_shrink
      n.spec.stretch = font_parameters.space_stretch
    else -- a glyph
      n = node.new("glyph")
      n.font = current_font
      n.subtype = 1
      n.char = s
      n.lang = tex.language
      n.uchyph = 1
      n.left = tex.lefthyphenmin
      n.right = tex.righthyphenmin
    end

    last.next = n
    last = n
  end

  -- now add the final parts: a penalty and the parfillskip glue
  local penalty = node.new("penalty")
  penalty.penalty = 10000

  local parfillskip = node.new("glue")
  parfillskip.spec = node.new("glue_spec")
  parfillskip.spec.stretch = 2^16
  parfillskip.spec.stretch_order = 2

  last.next = penalty
  penalty.next = parfillskip

  -- just to create the prev pointers for tex.linebreak
  node.slide(head)
  return head
end

local txt = "A wonderful serenity has taken possession of my entire soul, like these sweet mornings of spring which I enjoy with my whole heart. I am alone, and feel the charm of existence in this spot, which was created for the bliss of souls like mine."

tex.baselineskip = node.new("glue_spec")
tex.baselineskip.width = 14 * 2^16

local head = mknodes(txt)
lang.hyphenate(head)
head = node.kerning(head)
head = node.ligaturing(head)

local vbox
local size = 90
lines = 0
lines_goal = 6

while lines < lines_goal do
  texio.write_nl(string.format("Formatting text to %d mm",size))
  local copy_of_head = node.copy_list(head)
  vbox = tex.linebreak(copy_of_head,{ hsize = tex.sp(string.format("%dmm",size))})
  size = size - 10
  lines = count_lines(vbox)
  texio.write_nl(string.format("lines=%d",lines))
end

node.write(vbox)

\end{luacode*}
\end{document}

This creates a paragraph on the Lua side and re-typesets it until it is 6 lines (or longer). It does so by reducing the hsize from 90mm by 10mm on each iteration.

Answer 2

Here is a small context module that writes characters and spaces to a text file. Firstly, the Lua code that goes in a separate file charsperline.lua. It contains a simplistic callback (or “node finalizer” in Context lingo) that skims the hlists constituting paragraphs for characters and interword glue.

thirddata                  = thirddata or { }
thirddata.chars_per_line   = thirddata.chars_per_line or { }

local stringformat         = string.format
local tableconcat          = table.concat
local utfchar              = utf.char

local traverse_nodetype    = node.traverse_id
local traverse_nodelist    = node.traverse

local nodecodes            = nodes.nodecodes
local listcodes            = nodes.listcodes
local skipcodes            = nodes.skipcodes

local hlist_t              = nodecodes.hlist
local vlist_t              = nodecodes.vlist
local glue_t               = nodecodes.glue
local glyph_t              = nodecodes.glyph
local line_t               = listcodes.line
local userskip_t           = skipcodes.userskip

local tasks                = nodes.tasks
local enableaction         = tasks.enableaction
local disableaction        = tasks.disableaction

local linedata             = { }

local resolve_ligatures
resolve_ligatures = function (lst, hd)
  for n in traverse_nodetype (glyph_t, hd) do
    local components = n.components
    if components then
      lst = resolve_ligatures (lst, components)
    else
      lst[#lst+1] = utfchar (n.char)
    end
  end
  return lst
end

local collect = function (hd, groupcode)
  if groupcode == "vbox" then
    return hd
  end

  for current in traverse_nodetype (hlist_t, hd) do

    if current.subtype == line_t then
      local chars, has_glyphs = { }, false

      for n in traverse_nodelist (current.list) do

        local ntype, nsubtype = n.id, n.subtype

        -- we care only for glyphs’n’glue
        if ntype == glyph_t then
          has_glyphs = true
          if n.components then
            chars = resolve_ligatures (chars, n.components)
          else
            chars[#chars+1] = utfchar (n.char)
          end
        elseif ntype == glue_t and nsubtype == userskip_t then
          chars[#chars+1] = " "
        end

      end

      if has_glyphs then
        linedata[#linedata+1] = chars
      end
    end
  end

  return hd
end

thirddata.chars_per_line.collect = collect

tasks.appendaction ("finalizers", "before",
                    "thirddata.chars_per_line.collect")
tasks.disableaction("finalizers",
                    "thirddata.chars_per_line.count_words")

local write_stats = function (...) texiowrite_nl(stringformat(...)) end

local datafile = "./linedata.txt"

write_linedata = function (filename)
  filename = filename or datafile

  local result = { }
  for i = 1, #linedata do local line = linedata[i]
    result[#result+1] = stringformat ("%q,%d",
                                      tableconcat (line),
                                      #line)
  end

  io.savedata (filename, result, "\n")
end

local active --- callback state

commands.start_chars_per_line = function ()
  if not active then
    enableaction("finalizers",
                 "thirddata.chars_per_line.count_words")
    active = true
  end
end

commands.stop_chars_per_line = function ()
  if active then
    disableaction("finalizers",
                  "thirddata.chars_per_line.count_words")
    active = false
  end
end

commands.write_linedata = write_linedata

The user interface is defined in module t-charsperline.mkvi. Besides the usual \start... / \stop... environment it also sets up a call to write_linedata() at the end of the TeX run.

\startmodule [charsperline]

\unprotect

\ctxloadluafile{charsperline}

\def\startdumplines{\ctxcommand{start_chars_per_line ()}}

\def\stopdumplines{\endgraf\ctxcommand{stop_chars_per_line ()}}

\prependtoks \charsperline_dump \to \everystoptext

\def\charsperline_dump{\ctxcommand{write_linedata ()}}

\protect

\stopmodule \endinput

Now you can use the macros \startdumplines / \stopdumplines in regular documents by loading the module:

\usemodule[charsperline]
\setuplayout[width=5cm]

\starttext
  \startdumplines
    \input knuth
  \stopdumplines
\stoptext

The output is written to a file linedata.txt in the current directory. The schema for a line is CSV’ish: "<line content>",<character count>:

"Thus, I came to the conclu-",27
"sion that the designer of a",27
"new system must not only",24
"be the implementer and first",28
"large  scale user; the designer",31
"should also write the first user",32
"manual.",7
"The separation of any of these",30
"four components would have",26

Link to gist.

Btw. whether or not you get ASCII output depends on what glyphs a document contains; let me know if you desire a strict solution that strips all code points not in the ASCII range.