make4ht -ul with "pic-tabular"

Question

I am trying to generate a html file from a tex file containing Devanagari script in tabular by using

make4ht -ul book.tex "pic-tabular"

MWE for book.tex is:

\documentclass[11pt, oneside, onecolumn, openright, final]{article}

\usepackage{alternative4ht}
\usepackage{polyglossia}

\makeindex
\setmainlanguage[numerals=Devanagari]{hindi}
\setmainfont[Script=Devanagari, BoldFont={Sahadeva}]{Nakula}
\newfontfamily\devanagarifont[Script=Devanagari, BoldFont={Sahadeva}]{Nakula}


\begin{document}

\begin{tabular}{c}
  स
\end{tabular}

\end{document}

But the generated image and hence the html file contain a character X rather than स

It looks like utf8 is screwed somewhere, possible in

t4ht book.dvi 
....
Entering book.lg
System call: dvipng -T tight -D 144 -bg Transparent -pp 1:1 book.idv -o book0x.png

Is there a way to use dvipng or t4ht to handle unicode?

Any help in this regard is highly appreciated.

Thanks

Answer 1

This issue is not easy to fix. The well known issue is that tex4ht doesn't support OpenType fonts, they cause the compilation to fail. To overcome this issue, we use hacked versions of fontspec package with LuaLaTeX and XeLaTeX, whic blocks loading of fonts, the ordinary tfm fonts are used instead. To get Unicode character support we use different tricks with both LuaTeX and XeTeX, but in both cases we insert \special command which instrues tex4ht to replace the next character with Unicode value saved in the \special command. We use the "x" as this character, which will be eventually replaced.

The problem is that different DVI processors which are used when you generate pictures from the areas in the document don't unserstand tex4ht specials, so they just render this "x" character. Moreover, the font information is lost, so they wouldn't be able to render it correctly even if there was the correct Unicode character, as TFM fonts don't support that. And even if we used correct characters and correct fonts, not all DVI processors support OpenType fonts, just like tex4ht.

So, this seems a bit hopeless. Fortunatelly, I've found a solution, maybe bit complicated, but it seems to work:

• we can compile the document in the PDF mode using LuaLaTeX, together with tex4ht
• the configurations for fontspec must be suppressed in this mode, as we want OpenType fonts to load
• we provide special configuration to output each picture on it's own page
• we can insert Lua callback to find pages with pictures and save this information for the latter use
• we must suppress tex4ht dvi processing and image generation from DVI in this mode
• script which will process the pages with images will be executed, it will convert the picture pages to PNG or SVG using Ghostscript and pdf2svg

I've created a special make4ht build file, one LaTeX package with tex4ht configurations and few Lua libraries which do the hard work. Firs, the build file, sample.mk4:

local pdftoimg4ht = require "pdftoimg4ht"
local function write_empty_file(filename)
  local f = io.open(filename, "w")
  f:write("")
  f:close()
end

Make:add("pdftoimg", pdftoimg4ht.run)
Make:add("fakefontspec", function()
  -- block execution of these files
  write_empty_file("fontspec.4ht")
  write_empty_file("polyglossia.4ht")
  write_empty_file("usepackage-fontspec.4ht")
end)


set_settings {t4ht_par = "-p"}
Make:add("removefontspec", function()
  os.remove("fontspec.4ht")
  os.remove("usepackage-fontspec.4ht")
  os.remove("polyglossia.4ht")
end)

if mode=="images" then
  -- disable the default tex4ht support for fontspec
  Make:fakefontspec {}
  -- to suppress compilation error
  settings_add{ tex4ht_sty_par =  ",new-accents" }
  Make:htlatex {htlatex="lualatex", packages= "\\usepackage{save4htimages}"  }
  Make:pdftoimg {}
  -- disable the image conversion from t4ht
  Make:removefontspec {}
end

It defines a new mode, called images. In this mode, make4ht will produce the pictures, but not the HTML files. It must be called before the HTML generation. It can be executed in the following way:

 make4ht -e sample.mk4 -ulnm images book.tex "pic-tabular"

The -uln options require Unicode output, LuaLaTeX and disable tex4ht DVI processing, because it isn't necessary in the PDF mode. The -m images requires the "images" mode.

For the HTML generation, use

 make4ht -e sample.mk4 -ul book.tex "pic-tabular"

It is important to not use the -n and -m images options in this case.

Few additional files are necessary:

The save4htimages.sty:

\AtBeginDocument{%
  % Configure Picture commands to output it's contents on a new page
  \Configure{Picture+}{\newpage}{\newpage}
  \Configure{Picture*}{\newpage}{\newpage}
  % install Lua callbacks to remove tex4ht specials
  \directlua{
    local t4htcallback = require "fontspec4ht-images"
    luatexbase.add_to_callback("pre_linebreak_filter", t4htcallback.process, "remove tex4ht specials")
    luatexbase.add_to_callback("hpack_filter", t4htcallback.process, "remove tex4ht specials")
    luatexbase.add_to_callback("vpack_filter", t4htcallback.process, "remove tex4ht specials")
    % this callback will save the image pages
    luatexbase.add_to_callback("finish_pdffile", t4htcallback.save_pages, "save image pages")
  }
}

It configures tex4ht to place pictures on standalone pages and installs the callback which will save the picture page names and destination filenames. This information is saved in the \jobname-pagelist.lua file.

The callbacks are defined in the fontspec4ht-images.lua:

local M = {}
local hlist_id = node.id "hlist"
local vlist_id = node.id "vlist"
local whatsit_id = node.id "whatsit"
local glyph_id = node.id "glyph"
-- get the special subtype
local whatsits = node.whatsits()
local special_id  

local pagelist = {}


local utfchar = unicode.utf8.char
local function execute_tex4ht(head, n)
  local was_tex4ht = false
  local t4ht, data = n.data:match("(t4ht)(.+)")
  if t4ht == "t4ht" then was_tex4ht = true end
  if was_tex4ht then
    if data:match("@%+") then
      -- detect unicode characters
      local char = data:match("%{35%}x([0-9a-fA-F]+)%{59%}")
      if char then 
        -- we must replace the next glyph char with contents of this special
        local nextnode = n.next
        if nextnode.id == glyph_id then
          nextnode.char = tonumber(char, 16)
        end
      end
    elseif data:match("%+%+") then
      local picture_name = data:match("%+%+(.+)")
      -- sometimes we match something different than filename
      -- so try to detect that it is really a filename (we check that it ends
      -- with extension)
      if picture_name:match("%.[a-zA-Z]-$") then
        pagelist[picture_name] = tex.count[ "c@page" ]
      end
    end
  end
  return head, was_tex4ht
end

local function process(head)
  for n in node.traverse(head) do
    local id = n.id
    if id == hlist_id or id == vlist_id then
      n.head = process(n.head)
    elseif id == whatsit_id and (n.subtype == special_id or whatsits[n.subtype] == "special")  then
      special_id = n.subtype
      -- act on the special node and detect if it was tex4ht special
      local was_tex4ht 
      head, was_tex4ht= execute_tex4ht(head, n)
      if was_tex4ht then
        -- remove the special node
        head = node.remove(head, n)
      end
    end
  end
  return head
end

local function save_pages()
  local pagefile = tex.jobname .. "-pagelist.lua"
  local f = io.open(pagefile, "w")
  -- we will write the page list as Lua module consisting only from table
  local t = {"return {"}
  for k,v in pairs(pagelist) do
    t[#t+1] = string.format("[%s] = '%s',", v, k)
    -- print("save page", k, v)
  end
  t[#t+1] = "}"
  f:write(table.concat(t, "\n"))
  f:close()
end

M.process = process
M.pagelist = pagelist
M.save_pages = save_pages

return M

The images are generated using code from pdftoimg4ht.lua:

kpse.set_program_name "luatex"
local mkutils = require "mkutils"

local M = {}
local ghostscript = "gs"
-- output file resolution
local r = 72 * 4 
-- downscalling
local DownScaleFactor = 2 
local function get_bboxes(pdfname)
  -- the bounding box info is written to stderr, so we must redirect it to a temp file
  local tmpfile = pdfname .. "-tmp"
  local cmd = string.format("%s -q -sDEVICE=bbox -dNOPAUSE -dBATCH %s 2> %s", ghostscript, pdfname, tmpfile)
  print(cmd)
  local bboxes = {}
  os.execute(cmd)
  local executed = io.open(tmpfile, "r")
  local output = executed:read("*all")
  executed:close()
  os.remove(tmpfile)
  local page = 1
  -- find bounding boxes in the tmp file
  for x, y, x1, y1 in output:gmatch("%BoundingBox:%s*(%d+)%s*(%d+)%s*(%d+)%s(%d+)") do
    print(page, x,y,x1, y1)
    bboxes[page] = {x,y, x1, y1}
    page = page + 1
  end
  return bboxes
end

local function scale_point(point)
  return math.ceil(point * (r/72))
end
-- calculate dimensions for PDF cropping from the page bounding box
local function get_gs_dimension(x,y, x1, y1)
  local width = x1 - x
  local height = y1 - y
  return x, y, width, height
end

local function get_page_dimensions(bboxes, page)
  local bbox = bboxes[page] or {}
  return get_gs_dimension(bbox[1], bbox[2], bbox[3], bbox[4])
end

local function get_gs_page_options(bboxes, page)
  local x, y, width, height = get_page_dimensions(bboxes, page)
  print(x, y, width, height)
  return string.format('-r%d -dDownScaleFactor=%d -g%dx%d -c "<</Install {-%d -%d translate}>> setpagedevice" -dFirstPage=%d -dLastPage=%d', r, DownScaleFactor, scale_point(width)+1, scale_point(height)+1, x, y, page, page)
end

local function convert_png(filename, outputfile, bboxes, page)
  local options = get_gs_page_options(bboxes, page)
  local cmd = string.format("%s -q -sDEVICE=pngalpha -o %s %s %s", ghostscript, outputfile, options, filename)
  print(cmd)
  os.execute(cmd)
end

local function convert_svg(filename, outputfile, bboxes, page)
  local tmpname = os.tmpname() .. ".pdf"
  local options = get_gs_page_options(bboxes, page)
  local cmd = string.format("%s -q -sDEVICE=pdfwrite -o %s %s %s", ghostscript, tmpname, options, filename)
  print(cmd)
  os.execute(cmd)
  local pdf2svg = string.format("pdf2svg %s %s", tmpname, outputfile)
  print(pdf2svg)
  os.execute(pdf2svg)
  os.remove(tmpname)
end

local function convert_pagelist(filename, pagelist)
  local bboxes = get_bboxes(filename)
  for page, outputfile in pairs(pagelist) do
    if outputfile:match("png$") then
      convert_png(filename, outputfile, bboxes, page)
    elseif outputfile:match("svg$") then
      convert_svg(filename, outputfile, bboxes, page)
    else 
      print("unsupported output file format: ".. outputfile)
    end
  end
end

local function run(par)
  -- get options from the extension settings
  local ext_options = mkutils.get_filter_settings "pdftoimg4ht" or {}
  ghostscript = par.ghostscript or ext_options.ghostscript or ghostscript
  r = par.r or ext_options.r or r
  DownScaleFactor = par.DownScaleFactor or ext_options.DownScaleFactor or DownScaleFactor
  local pdffile = par.input .. ".pdf"
  local pagelist_file = par.pagelist_file or ext_options.pagelist_file or par.input .. "-pagelist.lua"
  local pagelist = require(pagelist_file)
  -- print(r, DownScaleFactor, pdffile, pagelist_file)
  convert_pagelist(pdffile, pagelist)
end




M.run = run
M.get_bboxes = get_bboxes
M.convert_png = convert_png
M.convert_svg = convert_svg
M.convert_pagelist = convert_pagelist

-- local pagelist = require "sample-pagelist"
-- convert_pagelist("sample.pdf", pagelist)

return M

It is possible to configure few things in Make:pdftoimg {} in the build file, for example the output resolution or command name for Ghostscript. For example the following may work on Windows:

Make:pdftoimg {ghostscript = "gswin32c"}

So this is it. I've modified your example for a better illustration of the conversion:

\documentclass[11pt, oneside, onecolumn, openright, final]{article}

\usepackage{alternative4ht}
\usepackage{polyglossia}

\makeindex
\setmainlanguage[numerals=Devanagari]{hindi}
\setmainfont[Script=Devanagari, BoldFont={Sahadeva}]{Nakula}
\newfontfamily\devanagarifont[Script=Devanagari, BoldFont={Sahadeva}]{Nakula}


\begin{document}

hello world

\begin{tabular}{c | c}
  \hline
  स & hello\\
  \hline
\end{tabular}

\end{document}

This is the result:

Btw, this proves that it is possible to use tex4ht in the PDF mode and the Lua callback can be modified to write the HTML and CSS files directly. So it was quite important experiment. I need to think how to incorporate all of this to make4ht and tex4ht.