2

Help me please,how i can insert pdf literal before and after paragraph node. My code works incorrect. Thanks everybody for the help.

\documentclass{article}
\usepackage{luacode}
\usepackage{polyglossia}
\setmainlanguage[babelshorthands=true]{russian}
\setmainfont{Times New Roman}
\pagestyle{empty}
\thispagestyle{empty}
\begin{luacode}
local pdf_node=node.new("whatsit","pdf_literal")
local pdf_node0=node.new("whatsit","pdf_literal")
pdf_node.data='/Span <</ActualText<FEFF041F04400438043204350442>>> BDC'
pdf_node0.data='EMC'
pdf_node.mode=2
pdf_node0.mode=2

local function insert_local_par(par,b)
par=node.insert_before(par,par,pdf_node)
par=node.insert_after(par,node.tail(par),pdf_node0)
end
luatexbase.add_to_callback("insert_local_par",insert_local_par,"insert_local_par")
\end{luacode}
\begin{document}
Test

hello
\newpage
par

new par
\end{document}
  • A local_par node is not a node list, so you cannot use insert_before and insert_after. Why don't you just use the accsupp package? That would probably be much easier. – Henri Menke May 14 at 5:35
  • @Henri Menk In future i want to do package,which automatic do tagging of pdf. may be someone have idea,how insert before and after paragraf pdf literal,or it impossible – Aleksandr Kozlovskiy May 14 at 15:00
2

Instead of doing it in a callback, you can examine the shipout box, searching for local_par.

\documentclass{article}
\usepackage{atbegshi}
\usepackage{luacode}
\begin{luacode}
local BDC = node.new("whatsit","pdf_literal")
local EMC = node.new("whatsit","pdf_literal")
BDC.data = '/Span <</ActualText<FEFF041F04400438043204350442>>> BDC'
BDC.mode = 2
EMC.data = 'EMC'
EMC.mode = 2

function tag_local_par(parent, level)
    local head = parent.list
    while head do
        -- texio.write_nl(string.rep("  ", level) .. tostring(head))
        if head.id == node.id"hlist" or head.id == node.id"vlist" then
            if head.list and head.list.id == node.id"local_par" then
                local par_list = head.list and head.list.next -- local_par should always be followed by a list
                head.list = node.insert_before(head.list, head.list, node.copy(BDC))
                par_list = node.insert_after(par_list, node.tail(par_list), node.copy(EMC))
            end
            tag_local_par(head, level + 1)
        end
        head = head.next
    end
end
\end{luacode}
\AtBeginShipout{\directlua{tag_local_par(tex.box["AtBeginShipoutBox"], 0)}}%
\begin{document}
Test

hello
\newpage
par

new par
\end{document}

Then, with pdftotext I get

Привет
Привет

1


Привет
Привет

2

A better tagging of paragraphs with their content can be done by processing the node list recursively like in https://tex.stackexchange.com/a/495230. Currently this takes place in the post_linebreak_filter callback, which is the correct callback to insert whatsits. However, at this point hyphenation, kerning, and linebreaking have all taken place and we end up with all kinds of details of these processes in the node list. That is why the output contains stray ligatures and hyphens, where there should be none.

The correct way to do this would be to convert scan the paragraphs in the hyphenate callback and them do some bookkeeping with node attributes or so and then later in the post_linebreak_filter collect all this information and place the /ActualText.

\documentclass{article}
\usepackage{luacode}
\begin{luacode}
local converters = {}

local function convert(n)
    local id = n.id
    local type = node.type(id)

    local typeconv = converters[type]
    if typeconv then
        return typeconv(n) or ""
    else
        texio.write_nl("tag_par warning: no conversion available for " .. type)
        return ""
    end
end

function converters.hlist(n)
    local text = {}
    for n in node.traverse(n.list) do
        text[#text + 1] = convert(n)
    end
    return table.concat(text)
end

function converters.glyph(n)
    return utf.char(n.char)
end

function converters.glue(n)
    -- FIXME: any glue is treated like space
    return " "
end

function converters.kern(n)
    -- FIXME: any kern is just dropped
    return ""
end

function converters.disc(n)
    -- FIXME: does anybody care about discretionaries? Can we even distinguish
    --        user and hyphenation ones?
    local subtype = node.subtypes(n.id)[n.subtype]
    if subtype == "automatic" then
        return convert(n.replace)
    end
    return ""
end

local function tag_par(head, groupcode)
    local text = {}
    for n in node.traverse(head) do
        text[#text + 1] = convert(n)
    end

    local actual_text = table.concat(text)
    actual_text = string.gsub(actual_text, " +", " ") -- collapse consecutive spaces
    actual_text = string.gsub(actual_text, "^%s*(.-)%s*$", "%1") -- trim surrounding spaces

    local BDC = node.new("whatsit", "pdf_literal")
    BDC.data = "/Span <</ActualText(<p>" .. actual_text .. "</p>)>> BDC"
    BDC.mode = 2
    head = node.insert_before(head, head, BDC)

    local EMC = node.new("whatsit", "pdf_literal")
    EMC.data = "EMC"
    EMC.mode = 2
    head = node.insert_after(head, node.tail(head), EMC)

    return head
end

luatexbase.add_to_callback("post_linebreak_filter", tag_par, "tag_par")

\end{luacode}
\begin{document}
\input{lorem.tex}

\input{knuth.tex}
\newpage
\input{ward.tex}

\input{zapf.tex}
\end{document}

For illustration purposes I have marked the paragraphs with HTML-like <p>...</p> tags. As can be seen from the output of pdftotext:

$ pdftotext test.pdf -
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur massa turpis, semper quis fringilla ut, viverra nec risus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Donec nunc lorem, sollicitudin vel sodales eget, vehicula nec mi. Proin ullamcorper rutrum nibh, at porttitor nunc euismod et. Donec faucibus nisi faucibus ipsum porttitor pharetra. Sed elementum, lectus nec congue imperdiet, ipsum leo viverra nisi, sit amet commodo odio odio id nisl. Fusce sagittis lobortis nisi sed consectetur. Nam egestas, sem ut fermentum convallis, ipsum tellus venenatis augue, eget condimentum risus quam id erat. Sed metus dui, sollicitudin pharetra pellen- tesque sed, placerat eget augue. Mauris sodales pretium tortor vitae rutrum. Proin quam sem, lobortis tincidunt pretium vitae, feugiat eu lacus.</p>

<p>Thus, I came to the conclusion that the designer of a new system must not only be the implementer and strst large||scale user; the designer should also write the strst user manual.</p>
<p>The separation of any of these four components would have hurt TEX signif- icantly. If I had not participated fully in all these activities, literally hundreds of improvements would never have been made, because I would never have thought of them or perceived why they were important.</p>

<p>But a system cannot be successful if it is too strongly in﬇uenced by a single person. Once the initial design is complete and fairly robust, the real test begins as people with many dierent viewpoints undertake their own experiments.</p>

1


<p>The Earth, as a habitat for animal life, is in old age and has a fatal illness. Several, in fact. It would be happening whether humans had ever evolved or not. But our presence is like the eect of an old|-|age patient who smokes many packs of cigarettes per day |=| and we humans are the cigarettes.</p>

<p>Coming back to the use of typefaces in electronic publishing: many of the new typographers receive their knowledge and information about the rules of typography from books, from computer magazines or the instruction manuals which they get with the purchase of a PC or software. There is not so much basic instruction, as of now, as there was in the old days, showing the dierences between good and bad typographic design. Many people are just fascinated by their PCâ•Žs tricks, and think that a widely||praised program, called up on the screen, will make everything automatic from now on.</p>

2
  • Thank you very mutch,but why we should use level variable? – Aleksandr Kozlovskiy May 15 at 16:03
  • @AleksandrKozlovskiy The level variable is not needed in this case, but I usually define it anyway because it is useful for tracing. I've added a commented line (texio.write_nl...) that prints the node tree as the list is traversed. That is very useful to verify that the new nodes are inserted in the right positions. – Henri Menke May 15 at 21:35
  • But in Thank you very mutch,but your code have an issue,i mean if line contains alot of chars,it not works correct. So,in your example pdf literal inserted only in the first line of each paragraf,but it must inserted at the start and at the end of each paragraf. – Aleksandr Kozlovskiy Jun 11 at 8:59
  • @AleksandrKozlovskiy Yes, you are right. The problem is that I insert EMC at the end of the hlist which is only the first line. I will fix this soon and update the answer. – Henri Menke Jun 11 at 11:43
  • Ok,i will wait. Thanks you very mutch for your help. – Aleksandr Kozlovskiy Jun 12 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.