10

Is possible to render paragraph in LuaTeX -> get ascii text with hyphenation -> put back text and render again?

  1. I have simple text: "Lorem ipsum dolor sit amet, conse... laborum."

I need get this text to render in 5 cm width in 12 pt specific font.

  1. render text and get array (with some data) Array

    temptext[0]="Lorem ipsum dolor sit a-"
    
    temptext[1]="met, consectetur adipisicing e-"
    
    temptext[2]="lit, sed do eiusmod tempor in-"
    
    ...
    
    temptext[n]="lit anim id est laborum."
    
  2. I change some parameters or letter for example new width is 5.1cm not 5.0cm, change some letter , change font etc.

  3. while my text have different lines than for example n=21 I change some parameters and go to 2.


This algorithm is possible to more than one file. for example my paragraph is \include file.tex and 3. point is run perl script, ruby, exe or bash script and run again. I need only number of lines , array table (what is first letter in line etc), and where are hyphenation mark.

Polish [Mogę napisać po polsku. Chodzi mi o to by dowiedziec się gdzie TeX wstawia znaki podziału wyrazu i tak dobrac parametry by wszystko zmieściło się w zadanym prostokącie (np. jakis skomplikowany tekst ze wzorem), zmieniając np. wielkosc fontu, glify, sam font itp.

Potrzebuje zatem informacji jak bardzo tekst nie pasuje do prostokąta. Gdzie ma błędy (overfull box), czyli gdzie wystaje, gdzie nie wystaje i o ile czyli end space size. Mam kilka glifów reprezentujących te same znaki. np. we wzorach moge użyć kilku fontów, które sa podobne do siebie, albo ligatury moge dorobić swoje byle wszystko pasowało do prostokatu akapitu. Alu musze wiedzieć jak bardzo nie pasuje to do akapitu. Tak by napisac program poprawiający tekst na tyle by błąd dopasowania był jak najmniejszy]

Jeśli ktoś może niech przetłumaczy polski tekst, który może jest jasniejszy na angielski. Dziękuję. ]

  • 1
    Here is some code that demonstrates how to extract the number of letters (and “words”) per line. If I understand you correctly for the kind of optimization you are attempting you don’t actually need the content of the lines, just n. Btw. I’m not sure there is an efficient solution to your problem as the breakpoints can vary a lot even with minuscule changes to a paragraph. – Philipp Gesang Aug 4 '13 at 17:57
  • condition is more complicated. not only number of line. In paragraph I have number paragraph and some number. I need for example all word on right have hyphenation on each line. etc. I need output format, no input words. I need information HOW TeX render, divide, change to ligatures , kerning etc. I need width line, strings, hyphen mark etc. Please think about it: I have many variables (font, size of font, bold or not tiltle etc.) and I put paragraph in to specyfic box (x,y). – Witold Aug 4 '13 at 18:08
  • @Witold: Then you don't want to somehow parse the result of the parbreaker, but interrogate the typesetting mechanism about its results (how many hpyhenations, how full are the lines etc.). – Reinstate Monica - M. Schröder Aug 7 '13 at 11:22
9

A proof of concept:

\documentclass{article}
\usepackage{luacode}
\usepackage{libertine}
\begin{document}
\begin{luacode*}
function count_lines (head)
  local linecount = 0
  while head do
    if head.id == 0 then linecount = linecount + 1 end
    head = head.next
  end
  return linecount
end

function mknodes( text )
  local current_font = font.current()
  local font_parameters = font.getfont(current_font).parameters
  local n, head, last
  -- we should insert the paragraph indentation at the beginning
  head = node.new("glue")
  head.spec = node.new("glue_spec")
  head.spec.width = 20 * 2^16
  last = head

  for s in string.utfvalues( text ) do
    local char = unicode.utf8.char(s)
    if unicode.utf8.match(char,"%s") then
      -- its a space
      n = node.new("glue")
      n.spec = node.new("glue_spec")
      n.spec.width   = font_parameters.space
      n.spec.shrink  = font_parameters.space_shrink
      n.spec.stretch = font_parameters.space_stretch
    else -- a glyph
      n = node.new("glyph")
      n.font = current_font
      n.subtype = 1
      n.char = s
      n.lang = tex.language
      n.uchyph = 1
      n.left = tex.lefthyphenmin
      n.right = tex.righthyphenmin
    end

    last.next = n
    last = n
  end

  -- now add the final parts: a penalty and the parfillskip glue
  local penalty = node.new("penalty")
  penalty.penalty = 10000

  local parfillskip = node.new("glue")
  parfillskip.spec = node.new("glue_spec")
  parfillskip.spec.stretch = 2^16
  parfillskip.spec.stretch_order = 2

  last.next = penalty
  penalty.next = parfillskip

  -- just to create the prev pointers for tex.linebreak
  node.slide(head)
  return head
end

local txt = "A wonderful serenity has taken possession of my entire soul, like these sweet mornings of spring which I enjoy with my whole heart. I am alone, and feel the charm of existence in this spot, which was created for the bliss of souls like mine."

tex.baselineskip = node.new("glue_spec")
tex.baselineskip.width = 14 * 2^16

local head = mknodes(txt)
lang.hyphenate(head)
head = node.kerning(head)
head = node.ligaturing(head)

local vbox
local size = 90
lines = 0
lines_goal = 6

while lines < lines_goal do
  texio.write_nl(string.format("Formatting text to %d mm",size))
  local copy_of_head = node.copy_list(head)
  vbox = tex.linebreak(copy_of_head,{ hsize = tex.sp(string.format("%dmm",size))})
  size = size - 10
  lines = count_lines(vbox)
  texio.write_nl(string.format("lines=%d",lines))
end

node.write(vbox)

\end{luacode*}
\end{document}

This creates a paragraph on the Lua side and re-typesets it until it is 6 lines (or longer). It does so by reducing the hsize from 90mm by 10mm on each iteration.

  • text must be a paragraph (trouble with unicode paragraph and math mode.) Is possible to write iteration to txt file? if I change lines_goal = 8 I have overfull error. Is possible to write iteration with error log? – Witold Aug 4 '13 at 20:30
  • 1
    @Witold What do you mean by "text must be a paragraph"? I am not sure if you can easily get the error messages, but you can get the total demerits from the line breaking function (see the documentation). Everything can be written to a file, of course. – topskip Aug 5 '13 at 7:52
  • text must be fit in a paragraph. (my error) I have paragraph in separately file.tex i can change file.tex and parameter of paragraph myself. But I canot know how fit my paragraph in boxes. How many lines i get, where is overfull box, and in which lines of text are to small space. Please look here imageshack.us/scaled/landing/196/s3iu.png and read my comment at end of the this page. I need better rendering than TeX. TeX cut my text to lines, some changes (add hyphen signs, change chars to ligatures etc.) I need information where is trouble. Where is white spaces or overfull box. – Witold Aug 5 '13 at 8:25
  • I have file.tex and script who tell me how my file.tex fit into paragraph. Line by line. Space by space. Next I change the file.tex (I change glyphs, change font in some letters, change spaces or some variats of chars) To better rendering paragraph. – Witold Aug 5 '13 at 8:29
  • Nicely done! For some reason I am getting a paragraph without kernings (seen around V in VLTAVA) and even ligatures (ffi in office) after I added these two words to the txt variable. Is there a new syntax? I am getting just some warnings from fontspec about missing features (WinXP,TL2014). – Malipivo Mar 24 '15 at 13:03
8

Here is a small context module that writes characters and spaces to a text file. Firstly, the Lua code that goes in a separate file charsperline.lua. It contains a simplistic callback (or “node finalizer” in Context lingo) that skims the hlists constituting paragraphs for characters and interword glue.

thirddata                  = thirddata or { }
thirddata.chars_per_line   = thirddata.chars_per_line or { }

local stringformat         = string.format
local tableconcat          = table.concat
local utfchar              = utf.char

local traverse_nodetype    = node.traverse_id
local traverse_nodelist    = node.traverse

local nodecodes            = nodes.nodecodes
local listcodes            = nodes.listcodes
local skipcodes            = nodes.skipcodes

local hlist_t              = nodecodes.hlist
local vlist_t              = nodecodes.vlist
local glue_t               = nodecodes.glue
local glyph_t              = nodecodes.glyph
local line_t               = listcodes.line
local userskip_t           = skipcodes.userskip

local tasks                = nodes.tasks
local enableaction         = tasks.enableaction
local disableaction        = tasks.disableaction

local linedata             = { }

local resolve_ligatures
resolve_ligatures = function (lst, hd)
  for n in traverse_nodetype (glyph_t, hd) do
    local components = n.components
    if components then
      lst = resolve_ligatures (lst, components)
    else
      lst[#lst+1] = utfchar (n.char)
    end
  end
  return lst
end

local collect = function (hd, groupcode)
  if groupcode == "vbox" then
    return hd
  end

  for current in traverse_nodetype (hlist_t, hd) do

    if current.subtype == line_t then
      local chars, has_glyphs = { }, false

      for n in traverse_nodelist (current.list) do

        local ntype, nsubtype = n.id, n.subtype

        -- we care only for glyphs’n’glue
        if ntype == glyph_t then
          has_glyphs = true
          if n.components then
            chars = resolve_ligatures (chars, n.components)
          else
            chars[#chars+1] = utfchar (n.char)
          end
        elseif ntype == glue_t and nsubtype == userskip_t then
          chars[#chars+1] = " "
        end

      end

      if has_glyphs then
        linedata[#linedata+1] = chars
      end
    end
  end

  return hd
end

thirddata.chars_per_line.collect = collect

tasks.appendaction ("finalizers", "before",
                    "thirddata.chars_per_line.collect")
tasks.disableaction("finalizers",
                    "thirddata.chars_per_line.count_words")

local write_stats = function (...) texiowrite_nl(stringformat(...)) end

local datafile = "./linedata.txt"

write_linedata = function (filename)
  filename = filename or datafile

  local result = { }
  for i = 1, #linedata do local line = linedata[i]
    result[#result+1] = stringformat ("%q,%d",
                                      tableconcat (line),
                                      #line)
  end

  io.savedata (filename, result, "\n")
end

local active --- callback state

commands.start_chars_per_line = function ()
  if not active then
    enableaction("finalizers",
                 "thirddata.chars_per_line.count_words")
    active = true
  end
end

commands.stop_chars_per_line = function ()
  if active then
    disableaction("finalizers",
                  "thirddata.chars_per_line.count_words")
    active = false
  end
end

commands.write_linedata = write_linedata

The user interface is defined in module t-charsperline.mkvi. Besides the usual \start... / \stop... environment it also sets up a call to write_linedata() at the end of the TeX run.

\startmodule [charsperline]

\unprotect

\ctxloadluafile{charsperline}

\def\startdumplines{\ctxcommand{start_chars_per_line ()}}

\def\stopdumplines{\endgraf\ctxcommand{stop_chars_per_line ()}}

\prependtoks \charsperline_dump \to \everystoptext

\def\charsperline_dump{\ctxcommand{write_linedata ()}}

\protect

\stopmodule \endinput

Now you can use the macros \startdumplines / \stopdumplines in regular documents by loading the module:

\usemodule[charsperline]
\setuplayout[width=5cm]

\starttext
  \startdumplines
    \input knuth
  \stopdumplines
\stoptext

The output is written to a file linedata.txt in the current directory. The schema for a line is CSV’ish: "<line content>",<character count>:

"Thus, I came to the conclu-",27
"sion that the designer of a",27
"new system must not only",24
"be the implementer and first",28
"large  scale user; the designer",31
"should also write the first user",32
"manual.",7
"The separation of any of these",30
"four components would have",26

Link to gist.

Btw. whether or not you get ASCII output depends on what glyphs a document contains; let me know if you desire a strict solution that strips all code points not in the ASCII range.

  • ! Undefined control sequence. l.1 \usemodule [charsperline] ? x No pages of output. BTW. What i can execute it? --shell-escape not working – Witold Aug 4 '13 at 20:34
  • You are right. I no need Ascii but number of glyphs (ligatures, glyphs etc.) and no count of words/characters but width of glypsh without spaces. I need overfull error and 'to small and to big space' in one line. I change the text or change the glyph if spaces is too big or text is not fit in width paragraph/box of the paragraph. – Witold Aug 4 '13 at 20:45
  • I need better rendering than standard. – Witold Aug 4 '13 at 20:58
  • 2
    This is Context code, you run it with context. (And you don’t need --shell-escape.) Can you state more precisely what you need? You appear to ask for all features of TeX nodes but as an array, which frankly doesn’t make much sense to me. As to the line badness (which causes the “overfull hbox” messages), you can get that as the second return value to node.hpack(). – Philipp Gesang Aug 4 '13 at 21:24
  • You code is running. Thanks. This code working with unicode, wonderfull. But... Please see You code: I get Overfull \hbox (23.36026pt too wide) in paragraph at lines 6--11 \3>modern-designsize-12pt-rm-tf-0--0 hun-dreds of im-prove-ments would ... this is overfull. === i use === \usemodule[charsperline] \setuplayout[width=5cm] \starttext \startdumplines Ąż już żółty \input knuth\par \stopdumplines \stoptext ===== i get ==== "Ąż już żółty Thus, I came to",28 "the conclusion that the de-",27 "signer of a new system must",27 "not only be the implementer",27 – Witold Aug 5 '13 at 6:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.