I am using pictureenv from the answer in tex4ht conflict with math in table vs. outside table when using SVG

When adding a second column to the table which contains listings, it changes the characters [ and _ anywhere in the listing to the letter x for some reason.

Here is a MWE




${\frac {d}{{d}x}}y \left( x \right) = \left( -2+x \right) ^{2}$&


Compiled with

 make4ht -ulm default -f html5+dvisvgm_hashes T.tex "htm,pic-m,pic-align,svg,p-width"

I also tried

 make4ht -ulm default -f html5+dvisvgm_hashes T.tex "htm,pic-tabular,pic-align,svg,p-width"


Mathematica graphics

Compare the output when the pictureenv is not used:




${\frac {d}{{d}x}}y \left( x \right) = \left( -2+x \right) ^{2}$&


Using same compile command, now it gives

Mathematica graphics

TL 2018 on Linux Ubuntu

1 Answer 1


What you see is the result of the tex4ht Unicode support. It works by inserting two codes to the document. First is special instruction which tells tex4ht to replace the next character with Unicode value stored in the instruction, the second one is character which will be replaced. It is usually x, but it can be any character. It is just used to pass the font information to tex4ht, so it can render the Unicode in bold, italics and so on.

The issue is with pictures, as they are generated by external commands, typically dvisvgm or dvipng. They don't know how to process tex4ht specials, so they will be ignored and only x is displayed.

We can try to fix this using LuaTeX. It is possible to process the document nodes using node callbacks, detect pictures and replace the characters by hand. It is not as easy as it sounds, because we cannot just set the Unicode value to the replaced character. It is necessary to set the correct glyph number instead. There is no universal mapping between Unicode and glyphs in the particular TeX fonts. Fortunatelly, tex4ht provide such mappings for most of the TeX fonts in the form of HTF files. It is possible to write Lua library which will search for the HTF files and parse it for the mappings.

This proved to be quite complicated stuff and I must admit that I've found a serious issue in the picture generation. Sometimes the mapping between the Unicode value and the font glyphs doesn't exist. For example, the \textellipsis command doesn't work even with this method. This shouldn't be problem in practise, as this limitation with pictures existed for quite some time and nobody ever complained. This is just a limitation I've found and I can't find a solution for now.

So this was enough introduction, we can go to the code now.

First, we need the HTF file library, htffontreader.lua:

kpse.set_program_name "luatex"
local entities = require "luaxml-entities"
local texmfdist = kpse.expand_var("$TEXMFDIST")
local default_paths = {
  texmfdist .. "/tex4ht/ht-fonts/mozilla/",
  texmfdist .. "/tex4ht/ht-fonts/unicode/",
  texmfdist .. "/tex4ht/ht-fonts/ascii/",
  texmfdist .. "/tex4ht/ht-fonts/alias/"

local function str_to_table(str)
  local characters = {}
  str:gsub(".", function(a) table.insert(characters, a) end)
  return characters

-- convert the .4ht string field to a Unicode codepoint
local function get_char(str)
  -- it is necessary to decode XML entites first
  local newstr = entities.decode(str)
  -- get Unicode codepoints of the string
  local chars = {}
  -- the string.utfvalues is LuaTeX extension 
  for codepoint in string.utfvalues(newstr) do
    chars[#chars+1] = codepoint
  -- return whole string if there is more than one codepoint
  -- it is useless in tex4ht char to node.char mapping
  if #chars > 1 then return newstr end
  return chars[1]

local function read_file(filename)
  local f = io.open(filename, "r")
  if not f then return nil, "Cannot open file " .. filename end
  local content = f:read("*all")
  return content

local function traverse_htf_files(dir, addresses)
  -- local addresses = addresses or {}
  for file in lfs.dir(dir) do
    -- skip current and parent dir links"
    if file ~= "." and file ~=".." then
      local current_path  =  dir .. file
      local attr = lfs.attributes(current_path)
      if attr.mode == "directory" then
        traverse_htf_files(current_path .. "/", addresses)
      elseif attr.mode == "file" then
        if file:match("htf$") then
          file = file:gsub(".htf$", "")
          -- print(current_path, attr.mode)
          addresses[file] = current_path
  return addresses

-- find all .htf and .4hf files in list of directories
local function find_htf_files(directories)
  local addresses = {}
  for _, dir in ipairs(directories) do
    addresses = traverse_htf_files(dir, addresses)
  return addresses

-- the htf files may contain only part of the font file name
-- we must build graph for efficient lookup for the correct
-- corresponding htf file
local function make_lookup_table(addresses)
  local function step(characters, lookup)
    if #characters > 0 then
      local char = table.remove(characters,1)
      local subtab = lookup[char] or {}
      lookup[char] = step(characters, subtab)
    return lookup
  local lookup = {}
  for file, _ in pairs(addresses) do
    -- get individual characters as a table
    local characters = str_to_table(file)
    lookup = step(characters, lookup)

  return lookup

local function lookup_font(font_name, lookup_table)
  local function lookup(characters, tbl)
    if #characters < 1 then return "" end
    local char = table.remove(characters, 1) 
    local subtab = tbl[char]
    if not subtab then return "" end
    return char .. lookup(characters, subtab)
  local characters = str_to_table(font_name)
  return lookup(characters, lookup_table)

local function get_htf_css(content)
  local htfcss = {}
  for name, style in content:gmatch("htfcss:%s*([%w]+)%s*([^\n]+)") do
    htfcss[name] = style
  return htfcss

local function parse_htf_line(line)
  -- details about the htf file: https://tug.org/applications/tex4ht/mn-htf.html
  -- from the manual:
  --   The ‘string’ field may include any sequence of characters, except for
  --   its delimiters. The backslash character ‘\’ acts there as an escaped
  --   character. It may act as a delimiter for a character code, or be
  --   followed by another backslash (that is, ‘\\’ represents the character
  --   ‘\’ ). 
  --   In the string part, use ‘&lt;’ for the character ‘<’, ‘&gt;’ for ‘>’, and ‘&amp;’ for ‘&’; 
  local escape = function(str)
    local str = str or ""
    str = str:gsub("\\\\", "\\"):gsub("\\'","'")
    return str
  local str, class = line:match("^%s*'(.-)'%s+'([0-9]*)'")
  -- from the manual: 
  --   A ‘class’ specified by an odd integer value asks for a
  --   pictorial character. An even integer number asks for a non-pictorial
  --   character, specified in the ‘string’ field. An empty class field is
  --   treated as a zero value. 
  if not str then return nil, "Cannot parse htf line: " .. line end
  class = class or "" -- add default value
  class = tonumber(class) or 0 -- convert empty class to zero
  return escape(str), class

local function parse_htf_glyphs(content, addresses)
  local map = {}
  local backmap = {}
  local readpos = 0
  local function readline()
    local start
    start, readpos, line = content:find("([^\n]-)\n", readpos)
    -- print(readpos, line)
    readpos = readpos + 1
    return line
  -- first detect if the htf file isn't only link to another one
  local link = content:match("^%s*%.([^%s]+)")
  if link then
    local newfile = addresses[link]
    if not newfile then return nil, "Cannot load htf file for ".. link end
    local content = read_file(newfile)
    return parse_htf_glyphs(content, addresses)
  -- read htf name, start char and end char
  local firstline = readline()
  local name, start, finish = firstline:match("^([^%s]+)%s+([%d]+)%s+([%d]+)")
  if not name then return nil, "cannot parse htf file" end
  -- convert the values to numbers
  local start, finish = tonumber(start), tonumber(finish)
  -- calculate number of lines to be read 
  local count = finish - start - 1
  for i = 1, count do
    local line = readline()
    -- char may be character code or list of character codes
    local str, class = parse_htf_line(line)
    local char = get_char(str) 
    -- print(start, line)
    -- print(start, str, class, char)
    -- map character code to the tfm font position
    if char then
      map[char] = start
    -- map tfm position to tex4ht character class and the replacement strin
    backmap[start] = {class = class, str = str}
    start = start + 1

  print("Parse htf font", name, start, finish)
  return map, backmap

local function load_font(font_name, addresses)
  --- todo: continue here
  local content, msg = read_file(font_name)
  if not content then return nil, msg end
  local htfcss = get_htf_css(content)
  -- return two tables, one from unicode to font positions, the other in the other direction
  local map, backmap = parse_htf_glyphs(content, addresses)
  return {htfcss = htfcss, map = map, backmap = backmap}

local function get_font(font_name, lookup_table, addresses)
  local htf_name = lookup_font(font_name, lookup_table)
  if htf_name and htf_name ~= "" then
    local font_file = addresses[htf_name]
    -- this shouldn't happen
    if not font_file then return nil, "Cannot find font file: " .. htf_name end
    return load_font(font_file, addresses)

    return nil, "Cannot find HTF font: " .. font_name

local function htfobject(paths)
  local paths = paths or default_paths
  local htfont = {}
  htfont.font_cache = {}
  htfont.addresses, msg = find_htf_files(paths)
  if not htfont.addresses then return nil, msg end
  htfont.lookup_table = make_lookup_table(htfont.addresses)
  function htfont:get_font(fontname)
    local f = self.font_cache[fontname] or get_font(fontname, self.lookup_table, self.addresses)
    self.font_cache[fontname] = f
    return f
  htfont.__index = htfont
  return setmetatable({}, htfont)

-- some testing
if arg[0] == "htffontreader.lua" then
  local htfx = htfobject()

  local cmsy = htfx:get_font("rm-lmr10")
  -- print(get_font("cmsy10", lookup_table, addresses))
  -- print(get_font("cmmi10", lookup_table, addresses))
  -- print(get_font("lm-ec1000", lookup_table, addresses))
  local cmss = htfx:get_font("cmss")
  for name, style in pairs(cmss.htfcss) do
    print(name, style)

local M = {}
M.htfobject = htfobject
return M

The picture processing callback is located in fixpictures4ht.lua library:

local htffontreader = require "htffontreader"
local hlist_id = node.id "hlist"
local vlist_id = node.id "vlist"
local whatsit_id = node.id "whatsit"
local glyph_id = node.id "glyph"
-- get the special subtype
local whatsits = node.whatsits()
local special_id  

-- font database object
local fontdb = htffontreader.htfobject()

local supported_htf_fonts

-- from Luaotfload documentation
local function unsafe_getfont (id)
  local tfmdata = font.getfont (id)
  if not tfmdata then
    tfmdata = font.fonts[id]
  return tfmdata

local font_infos = {}
local function get_font_info(id)
  local info = font_infos[id]
  if info then return info end
  local tfmdata = unsafe_getfont(id)
  local name = tfmdata.name
  local format = tfmdata.properties.format
  font_infos[id] = name
  print("Loading htf file for " .. name)
  return name

local utfchar = unicode.utf8.char
local in_picture = false
local function execute_tex4ht(head, n)
  local was_tex4ht = false
  local t4ht, data = n.data:match("(t4ht)(.+)")
  if t4ht == "t4ht" then was_tex4ht = true end
  if was_tex4ht then
    if in_picture then
      -- tex4ht.sty definition for the \Picture(+|*) commands redefines the \ht:special command to propend t4ht+ in fornt of 
      -- the special code. I guess that the tex4ht command then somehow handles that, but I didn't investigate that. anyway, 
      -- we need to remove the spurious +t4ht part
      data = data:gsub("^%+t4ht","")
    if in_picture and data:match("^@") then
      -- interpolate tex4ht escaped entities
      data = data:gsub("{([0-9]+)}", function(x) return string.char(x) end)
      -- detect hexadecimal entities
      local char = data:match("%&%#x([0-9a-fA-F]+);") 
      if char then
        char = tonumber(char, 16)
        -- decimal entity
        char = data:match("^@([0-9]+)") or data:match("^@%&%#([0-9]+;")
        if char then
          char = tonumber(char)
      if char then 
        -- we must replace the next glyph char with contents of this special
        local nextnode = n.next
        if nextnode.id == glyph_id then
          -- it is necessary to do new kerning
          local font_name = get_font_info(nextnode.font)
          local fontdata = fontdb:get_font(font_name)
          local nextchar = fontdata.map[char]
          if nextchar then
            nextnode.char = nextchar
            -- the character is not available in the htf file. why?
            -- one possibility is the non breaking space
            if char == 160 or char==32 then
              -- replace it with ordinary space?
              local glue = node.new("glue")
              glue.width = tex.sp(".6em")
              n.next = glue
              glue.next = nextnode.next
        print("data", data)
    elseif data:match("%+%+") then
      local picture_name = data:match("%+%+(.+)")
      -- sometimes we match something different than filename
      -- so try to detect that it is really a filename (we check that it ends
      -- with extension)
      if picture_name:match("%.[a-zA-Z]-$") then
        print("start picture", picture_name)
        in_picture = true
        -- pagelist[picture_name] = tex.count[ "c@page" ]
    elseif data == "+" then
      print "end picture"
      in_picture = false
  return head, was_tex4ht

local function process(head)
  for n in node.traverse(head) do
    local id = n.id
    if id == hlist_id or id == vlist_id  then
      n.head = process(n.head)
    elseif id == whatsit_id and (n.subtype == special_id or whatsits[n.subtype] == "special")  then
      special_id = n.subtype
      -- act on the special node and detect if it was tex4ht special
      local was_tex4ht 
      head, was_tex4ht= execute_tex4ht(head, n)
  return head

local M = {}
M.process = process
return M

The callback must be installed, which can be done in redefined version of tuenc-luatex.4ht file:

% tuenc-luatex.4ht, generated from tex4ht-4ht.tex
% Copyright 2017 TeX Users Group
% This work may be distributed and/or modified under the
% conditions of the LaTeX Project Public License, either
% version 1.3c of this license or (at your option) any
% later version. The latest version of this license is in
%   http://www.latex-project.org/lppl.txt
% and version 1.3c or later is part of all distributions
% of LaTeX version 2005/12/01 or later.
% This work has the LPPL maintenance status "maintained".
% This Current Maintainer of this work
% is the TeX4ht Project <[email protected]>.
% If you modify this program, changing the
% version identification would be appreciated.
\immediate\write-1{version 2017-01-24-15:21}


  local fontspec = require "fontspec-4ht"
  local fixfonts = require "fixpictures4ht"
  luatexbase.add_to_callback("pre_linebreak_filter", fontspec.char_to_entity, "Char to entity")
  luatexbase.add_to_callback("hpack_filter", fontspec.char_to_entity, "hpack-char-to-entity")
  luatexbase.add_to_callback("pre_linebreak_filter", fixfonts.process, "Fix unicode in pictures")

There is also issue that the default configurations for listings are quite involved and redefine lot of stuff. You don't want it in the picture mode, so we must configure the pictureenv environment to ignore most of that:


The configuration


is especially important for multiline listings, because the default configuration causes them to collapse to just one line.

I've prepared an example with more illustrations:




${\frac {d}{{d}x}}y \left( x \right) = \left( -2+x \right) ^{2}$&

divný příliš~žluťoučký kůň \textunderscore 
  some spaces
no spaces

divný příliš~žluťoučký kůň \textunderscore 




This is the default rendering (without listings configurations!):

enter image description here

This is the result processed with fixpictures4ht.lua:

enter image description here

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .