2

Please help me to create a regular expression, which, for example, replaces $a+b$ with \(a+b\) and, for example, $$a-b$$ with \[a-b\], such that I can then hook into the \(...\) and \[...\] macros to annotate the contained material with /ActualText for accessibility. The replacement must be only happen, if before each $ there is no \ char.

Here is my code, which does it. For $$ it works very well, but for $ I have errors, because in some places $ is replaced with \( or, maybe, \) even if before $ there is the \ char.

\documentclass{article}
\begin{document}
\directlua{require("test.lua")}
test
summ
\$$$a=\sum_{i=0}^n{a_i^2}+1\$$$ \$$a-b=0$\$
\[a=\sum_{i=0}^n{a_i^2}-1\] \(a=\sum_{i=0}^n{a_i^2}\)
\end{document}

test.lua

function process_input_buffer(buffer)
texio.write_nl("callback get line "..buffer)
buffer=buffer:gsub("([^\\]?)%$%$(.-)([^\\]?)%$%$","%1\\[%2%3\\]")
texio.write_nl("callback get line "..buffer)
buffer=buffer:gsub("([^\\]?)%$(.-)([^\\]?)%$","%1\\(%2%3\\)")
texio.write_nl("callback get line "..buffer)
return buffer
end
luatexbase.add_to_callback("process_input_buffer",process_input_buffer,"buffer")
  • 3
    you ask for a regular expression but you seem to be using Lua patterns which are a lot less expressive than regular expressions. – David Carlisle Jun 9 at 21:25
  • 2
    Lua Patterns have some slight similarities with regular expressions but they are a completely different system. Also why do this in a callback, changing $ to \( in a callback just makes things slow, Lua has to change $ to \( then tex replaces \( by $ to start math mode. the benefits of using \( rather than $ are all about having matching delimiters in the file to help editor highlighting etc, – David Carlisle Jun 9 at 21:45
  • 2
    And why don't you just change the input then? There are enough tools around which can change a file based on regular expressions... – Skillmon Jun 9 at 22:11
  • 2
    That is a pretty tough one because you can have nested $, e.g. $a + b * \text{d $e$ f} + g$. – Henri Menke Jun 10 at 7:39
  • 1
    As Henri just observed nested $ expressions make this also impossible to do with regex (you can not match brackets in true regular expressions, you need an extended system that can count) but note I did not say that you could not do this with Lua patterns I just observed that your question was (and still is) inconsistent as it asks in the text for help with a regular expression but it shows an example of Lua patterns. Nie Lua can count so if you really needed this you could march on each $ separately and then decide the action in a Lua function – David Carlisle Jun 10 at 20:09
5

Rather than trying to process the document source and having to fight with all kinds of problems like catcodes, it would be much easier to tag formulas with /ActualText in the mlist_to_hlist callback.

In a private email Ulrike told me that you want to tag math for accessibility purposes. Below I present my humble solution.

Features

  1. Inside the callback the mlist is walked linearly (not recursively) and the convert function is called for every node in the list.

  2. The convert function inspects the node's type and looks up the appropriate conversion function in the table converters. If no converter is available, a warning is printed in the log and an empty string is returned, i.e. the node is ignored.

  3. The conversion functions typically just call convert on each of their fields and concatenate the result in a sensible manner. Conversion only stops, when an ignored node is encountered or the node has type math_char, because there we know what to do.

  4. For nodes of type math_char we extract the character. The character is stored as the number of the slot in the font. That is why we require unicode-math, otherwise the result is too ambiguous because it depends on the font. For Unicode characters we can either emit the Unicode character directly or we can use the table of Unicode math symbols to look up the command to produce that symbol and emit this.

Limitations

  1. The code below is only an example. Many node types are not handled, especially the sub_box type, so embedded horizontal and vertical mode material is simply lost.

  2. Nested mlists currently fail badly. I think this is a limitation of /ActualText, not being able to be nested.

  3. Fences are just output as characters with \left and \right. Manually scaled delimiters are fences nested in sub_box nodes, which are not handled currently.

  4. In the source file we use one math operator \Res. This currently outputs as Res, i.e. the markup is entirely lost. This is actually a very tricky one, because what \Res essentially does is \mathop{<switch to text font but stay in math mode> Res}. The operation of switching to the text font is in principle encoded in the math_char nodes' fam field, but not handled right now.

  5. More limitations are marked in the code with FIXME.

You can also find this code with some other random improvements on my GitHub:
https://gist.github.com/hmenke/a41574ef0b5000635986f0dec73e066f

tag_math.lua

local unimath_symbols = {}
local f = io.open(kpse.find_file("unicode-math-table.tex"), "r")
for line in f:lines() do
    local slot, cmd = string.match(line, [[^\UnicodeMathSymbol{"([%a%d]*)}{([^}%s]*)%s*}]])
    if slot then
        unimath_symbols[tonumber(slot, 16)] = cmd
    end
end
f:close()

local function convert_char(c)
    return unimath_symbols[c] or utf.char(c)
end

local converters = {}

local function convert(n)
    local id = n.id
    local type = node.type(id)

    local typeconv = converters[type]
    if typeconv then
        return typeconv(n) or ""
    else
        texio.write_nl("tag_math warning: no conversion available for " .. type)
        return ""
    end
end

function converters.noad(n)
    if not (n.nucleus.head or n.nucleus.char) then
        -- This is a thing, e.g. ${}$ is just an empty noad
        return ""
    end
    local result = convert(n.nucleus)

    local subtype = node.subtypes(n.id)[n.subtype]
    if subtype == "oplimits" or subtype == "opdisplaylimits" then
        result = result .. "\\limits"
    end

    if n.sub then
        result = result .. "_{" .. convert(n.sub) .. "}"
    end
    if n.sup then
        result = result .. "^{" .. convert(n.sup) .. "}"
    end

    return result
end

function converters.math_char(n)
    return convert_char(n.char)
end

function converters.sub_mlist(n)
    local result = ""
    for n in node.traverse(n.head) do
        result = result .. convert(n)
    end
    return result
end

function converters.fence(n, subtype)
    local subtype = node.subtypes(n.id)[n.subtype]
    local leftright = { left = "\\left", right = "\\right" }

    local result
    if n.delim.small_char ~= 0 then
        result = convert_char(n.delim.small_char)
    elseif n.delim.large_char ~= 0 then
        result = convert_char(n.delim.large_char)
    else
        result = "."
    end

    return leftright[subtype] .. result
end

function converters.fraction(n)
    local num = convert(n.num)
    local denom = convert(n.denom)
    return "\\frac{" .. num .. "}{" .. denom .. "}"
end

function converters.radical(n)
    local result = "\\sqrt{" .. convert(n.nucleus) .. "}"

    if n.sub then
        result = result .. "_{" .. convert(n.sub) .. "}"
    end
    if n.sup then
        result = result .. "^{" .. convert(n.sup) .. "}"
    end

    return result
end

function converters.style(n)
    return "\\" .. n.style .. "style"
end

function converters.accent(n)
    local result = convert(n.nucleus)
    if n.accent then
        result = convert(n.accent) .. "{" .. result .. "}"
    end
    if n.bot_accent then
        result = convert(n.bot_accent) .. "{" .. result .. "}"
    end
    if n.sub then
        result = result .. "_{" .. convert(n.sub) .. "}"
    end
    if n.sup then
        result = result .. "^{" .. convert(n.sup) .. "}"
    end
    return result
end

function converters.glue(n)
    -- FIXME: any glue is treated like space
    return " "
end

function converters.kern(n)
    -- FIXME: any kern is just dropped
    return ""
end

local function tag_math(head, display_type, need_penalties)
    local text = {}
    for n in node.traverse(head) do
        text[#text + 1] = convert(n)
    end

    -- concatenate, escape, and remove quotes
    local actual_text = string.sub(string.format("%q", table.concat(text, "")), 2, -2)
    if display_type == "display" then
        actual_text = "\\\\[" .. actual_text .. "\\\\]"
    elseif display_type == "text" then
        actual_text = "\\\\(" .. actual_text .. "\\\\)"
    end

    local BDC = node.new("whatsit", "pdf_literal")
    BDC.data = "/Span <</ActualText(" .. actual_text .. ")>> BDC"
    BDC.mode = 2
    head = node.insert_before(head, head, BDC)

    local EMC = node.new("whatsit", "pdf_literal")
    EMC.data = "EMC"
    EMC.mode = 2
    head = node.insert_after(head, node.tail(head), EMC)

    return node.mlist_to_hlist(head, display_type, need_penalties)
end

luatexbase.add_to_callback("mlist_to_hlist", tag_math, "tag_math")

test.tex

\documentclass{article}
\pagestyle{empty}
\usepackage{amsmath}
\usepackage{unicode-math}
\DeclareMathOperator\Res{Res}
\AtBeginDocument{\directlua{dofile("tag_math.lua")}}
\begin{document}

$
    \frac{1}{2\pi i} \int\limits_\gamma f\left(x^{\symbf{N}\in\mathbb{C}^{N\times 10}}\right)
    = \sum_{k=1}^m n(\gamma;a_k)\Res(f;a_k)\,.
$

\[
    \frac{1}{2\pi i} \int\limits_\gamma f\left(x^{\symbf{N}\in\mathbb{C}^{N\times 10}}\right)
    = \sum_{k=1}^m n(\gamma;a_k)\Res(f;a_k)\,.
\]

\end{document}

original output

Running pdftotext on the output I get

$ pdftotext test.pdf -
\(\frac{1}{2\mitpi\miti}\int\limits_{\mitgamma}\mitf\left\lparen\mitx^{\mbfN\in\BbbC^{\mitN\times10}}\right\rparen\equal\sum\limits_{\mitk\equal1}^{\mitm}\mitn\lparen\mitgamma\mathsemicolon\mita_{\mitk}\rparenRes\lparen\mitf\mathsemicolon\mita_{\mitk}\rparen \mathperiod\)
\[\frac{1}{2\mitpi\miti}\int\limits_{\mitgamma}\mitf\left\lparen\mitx^{\mbfN\in\BbbC^{\mitN\times10}}\right\rparen\equal\sum\limits_{\mitk\equal1}^{\mitm}\mitn\lparen\mitgamma\mathsemicolon\mita_{\mitk}\rparenRes\lparen\mitf\mathsemicolon\mita_{\mitk}\rparen \mathperiod\]

This is almost valid TeX code and it almost round-trips.

In this example one has to fix \rparenRes to \rparen\opertorname{Res}, but then it gives almost identical output (apart from \int which is missing \limits).

\documentclass{article}
\pagestyle{empty}
\usepackage{amsmath}
\usepackage{unicode-math}
\begin{document}
\(\frac{1}{2\mitpi\miti}\int\limits_{\mitgamma}\mitf\left\lparen\mitx^{\mbfN\in\BbbC^{\mitN\times10}}\right\rparen\equal\sum\limits_{\mitk\equal1}^{\mitm}\mitn\lparen\mitgamma\mathsemicolon\mita_{\mitk}\rparen\operatorname{Res}\lparen\mitf\mathsemicolon\mita_{\mitk}\rparen \mathperiod\)
\[\frac{1}{2\mitpi\miti}\int\limits_{\mitgamma}\mitf\left\lparen\mitx^{\mbfN\in\BbbC^{\mitN\times10}}\right\rparen\equal\sum\limits_{\mitk\equal1}^{\mitm}\mitn\lparen\mitgamma\mathsemicolon\mita_{\mitk}\rparen\operatorname{Res}\lparen\mitf\mathsemicolon\mita_{\mitk}\rparen \mathperiod\]
\end{document}

round-tripped output

  • Imho the OP wants to get the tex code of the math inside the /Actualtext. – Ulrike Fischer Jun 11 at 7:35
  • @UlrikeFischer But then it's just a duplicate of tex.stackexchange.com/questions/452532/… – Henri Menke Jun 11 at 9:19
  • @henri-menke,@Ulrike Fischer Yes,i want to se latex notation of formula,or another notation,but it must be show text of formula in actualtext. Also i not understoode,how i can disable callback in equation and enable it again. – Aleksandr Kozlovskiy Jun 11 at 9:38
  • @AleksandrKozlovskiy You have to make a choice. It is either catcode stability and no balancing issues using the callback or no verbatim material and fragile $ counting using something like in tex.stackexchange.com/questions/452532/…. You can't have both. – Henri Menke Jun 11 at 9:49
  • @Henri Menke but how get text of formula from node,which give this callback. – Aleksandr Kozlovskiy Jun 11 at 10:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.