6

I would like to do syntax highlighting with lua but I am struggling with getting the listing content into lua. As far as I understand the \_prepareverbdata macro can be overwritten to inject color changing macros. This is what I have so far:

\beglua
function highlight (x)
    -- The idea is to use a library here to do the actual syntax highlighting.
    tex.print("\\Blue " .. x)
end
\endlua

\def\_prepareverbdata#1#2{%
    \edef#1{\directlua{highlight("\luaescapestring{#2}")}}%
}

\begtt
fn main() {
    println!("Hello, world!");
}
\endtt

\bye

The above code results in this:

screenshot of result of above tex code

How can I get the listing content into lua and inject color changing macros?

2 Answers 2

7

There are two problems with the current code.

The space character inside verbatim text is active, and is defined as the primitive \ (or to be more precise, to an equivalent macro called \_dsp). This primitive is not expandable. Everything inside a \directlua is getting expanded as much as it can before being moved to the lua interpreter , so all the spaces are expanded to \_dsp, then the lua interpreter use it as a string, and when you print it back to TeX the tokens that are printed are using the current category code regime, and in verbatim \ has category code 12. To prevent that from happening you can stop the expansion early (which you should probably do anyway dealing with verbatim) using \unexpanded.

The second problem is that when you print \Blue from lua the current catcode regime is being used, and as before \ has category code 12. You can use tex.tprint to print \Blue in a different catcode regime (using the \optexcatcode catcode table for example). You might wonder why

\def\_prepareverbdata#1#2{%
    \def#1{\Blue #2}%
}

works as expected. This is because TeX is tokenizing at definition time.

I used \Red because it is more visible in the picture

\beglua
function highlight (x)
    -- The idea is to use a library here to do the actual syntax highlighting.
    tex.tprint({optex.registernumber'_optexcatcode',"\\Red "}, {x})
end
\endlua

\def\_prepareverbdata#1#2{%
    \edef#1{\directlua{highlight("\luaescapestring{\unexpanded{#2}}")}}%
}

\begtt
fn main() {
    println!("Hello, world!");
}
\endtt

\bye

enter image description here

Note that there are ways to scan arguments and push back from lua to TeX without these kind of troubles. See the token library for example. In this example can simply set the correct attribute, so you can do the following instead of printing the color macro

{\localcolor\Red} % making sure that the attribute for Red was defined
\beglua
local color_attr = optex.registernumber('_colorattr')
local attr_val = tonumber(token.get_macro('_color::0 1 1 0 k'))
function highlight (x)
    -- The idea is to use a library here to do the actual syntax highlighting.
    tex.setattribute(color_attr,attr_val)
    tex.print(x)
end
\endlua

\def\_prepareverbdata#1#2{%
    \edef#1{\directlua{highlight("\luaescapestring{\unexpanded{#2}}")}}%
}

\begtt
fn main() {
    println!("Hello, world!");
}
\endtt

\bye
4
  • Alternatively, you could use optex.set_node_color to do the colouring, but you'd need to use node.write instead of tex.print to output the characters for it to work properly. Commented Jun 16 at 8:48
  • 1
    @MaxChernoff That is a good point. It really depends on what is the intention in The idea is to use a library here to do the actual syntax highlighting. I guess.
    – Udi Fogiel
    Commented Jun 18 at 19:29
  • @MaxChernoff that seems like the cleanest option however I can't figure out how node.write works. Can you point me in the right direction?
    – pinsl
    Commented Jun 24 at 12:28
  • @pinsl You need to convert the string into a node list first; something like tex.forcehmode(); for _, c in utf8.codes("abcdefg") do; local n = node.new("glyph"); n.font = font.current(); n.char = c; node.write(n); end should work, but it's even easier if you can let TeX create the node list for you. I've got a full demonstration in my answer. Commented Jun 25 at 5:57
3

Instead of outputting colour-changing macros, we can scan in the verbatim text with token.scan_list(), change the colour of the glyph nodes with optex.set_node_color(), then write them to the page with node.write(). For this example to work, you'll need to download lexer.lua and rust.lua from the Scintillua project and place them in the same folder as your TeX file.

\fontfam[lm]
\nopagenumbers
\hyperlinks \Blue \Blue

\beglua
    -- Bug fix for am internal OpTeX function
    local SPACE = utf8.codepoint(" ")
    local original_set_node_color = optex.set_node_color

    function optex.set_node_color(...)
        local space_catcode = tex.catcode[SPACE]
        tex.catcode[SPACE] = 12
        original_set_node_color(...)
        tex.setcount("global", "_colorcnt", tex.count._colorcnt)
        tex.catcode[SPACE] = space_catcode
    end

    -- Save some locals
    local concatenate   = table.concat
    local force_hmode   = tex.forcehmode
    local remove_node   = node.remove
    local set_colour    = optex.set_node_color
    local table_insert  = table.insert
    local table_remove  = table.remove
    local traverse      = node.traverse
    local utf_char      = utf8.char
    local write_node    = node.write

    -- Define some constants
    local GLUE_ID          = node.id("glue")
    local GLYPH_ID         = node.id("glyph")
    local MAX_INTEGER      = math.maxinteger

    -- Define the colour scheme in HTML hex format
    local colours = {
        ["comment"]          = {0x66, 0x66, 0x66},
        ["default"]          = {0x00, 0x00, 0x00},
        ["function.builtin"] = {0xCC, 0x66, 0x00},
        ["function"]         = {0x1A, 0x1A, 0x1A},
        ["identifier"]       = {0x1A, 0x1A, 0x1A},
        ["keyword"]          = {0x00, 0x66, 0xCC},
        ["number"]           = {0x00, 0x99, 0x99},
        ["operator"]         = {0x1A, 0x1A, 0x1A},
        ["preprocessor"]     = {0xCC, 0x00, 0x66},
        ["string"]           = {0x00, 0x99, 0x00},
        ["type"]             = {0x66, 0x00, 0xCC},
        ["whitespace.rust"]  = {0x1A, 0x1A, 0x1A},
    }

    -- Convert the colours to PDF format
    local pdf_formatter = string.formatters["%.2f %.2f %.2f rg"]
    for name, colour in pairs(colours) do
        colours[name] = pdf_formatter(
            colour[1] / 255, colour[2] / 255, colour[3] / 255
        )
    end

    -- Load Scintillua
    local scintillua = require("lexer")
    local rust_lexer = scintillua.load("rust")

    -- Define the highlighting functions
    local function get_next_colour(lexed)
        local this_colour, next_pos

        if #lexed >= 2 then
            local colour_name = table_remove(lexed, 1)
            this_colour = colours[colour_name]
            if not this_colour then
                this_colour = colours["default"]
            end

            next_pos = table_remove(lexed, 1)
        else
            this_colour = colours["default"]
            next_pos = MAX_INTEGER
        end

        return this_colour, next_pos
    end

    -- Define the TeX command
    optex.define_lua_command("_luaprintverb", function()
        -- Grab the text
        local hlist = token.scan_list().list

        if not hlist then
            force_hmode()
            return
        end

        -- Convert all the nodes into a string
        local chars = {}

        for n in traverse(hlist) do
            local char
            if n.id == GLYPH_ID then
                char = utf_char(n.char)
            elseif n.id == GLUE_ID then
                char = " "
            end

            if char then
                table_insert(chars, char)
            else
                hlist = remove_node(hlist, n)
            end
        end

        local text = concatenate(chars)

        -- Lex the text
        local lexed = rust_lexer:lex(text)

        -- Highlight all the nodes
        local i = 0
        local this_colour, next_pos = get_next_colour(lexed)

        for n in traverse(hlist) do
            i = i + 1

            if i >= next_pos then
                this_colour, next_pos = get_next_colour(lexed)
            end

            set_colour(n, this_colour)
        end

        -- Output the text
        force_hmode()
        write_node(hlist)
    end)
\endlua

\def\_printverbline#1{\_luaprintverb\_hbox{#1}\_par}

From \url{https://doc.rust-lang.org/rust-by-example/hello/print/print_debug.html}:
\begtt
#[derive(Debug)]
struct Person<'a> {
    name: &'a str,
    age: u8
}

fn main() {
    let name = "Peter";
    let age = 27;
    let peter = Person { name, age };

    // Pretty print
    println!("{:#?}", peter);
}
\endtt

\bye

output

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .