Using direct lua to read binary bytes to plain text?

Question

In this thread, @David Carlisle tells us how to use luatex to read some of the properties of a png image. Alas, the image library seems to fail regarding the colorspace property. Always reports nil.

If I could read two bytes from the file (at known locations), and have those bytes assigned to a LaTeX variable as plain text 00 through FF, then I know what to do from there.

I've looked at this thread, which seems related, but it does not provide what I need.

So, here is my general question: How would Lua code do this:

1) Open an arbitrary binary file (partially). Although I have PNG in mind, I assume that any file type could be used. The method must be cross-platform.

2) Select only the bytes from offset P to offset Q, where P and Q are provided as variables, via a LaTeX macro. Might be as much as a couple of hundred bytes, but probably much less.

3) Express the bytes as a plain text string, using codes 0 through F. So, if asked for ten bytes, the string might look something like A9003405BC without space or return.

4) Assign the string to a LaTeX macro, for subsequent processing via ordinary LaTeX.

Why I want to know: In my LuaLaTeX application, the use of images is limited. I understand the image specs, but (not being a programmer) am baffled by such things as the LuaTeX manual.

Why anyone else should care: Seems to me that the power of Lua is under-used in LuaLaTeX, probably because most users are doing a one-time project, and don't want to learn another programming language. My request is for something that is modular, and has general applicability.

Note that a solution to 1-4 also can be used when the desired information is not at a specific byte offset, as long as it is somewhere near the start of the file. Then, the plain text string can be parsed for values, using ordinary LaTeX methods.

EDIT: Looking at David's answer, below, I have some questions:

The desired code is something like this (PSEUDOCODE, THIS DOES NOT WORK):

\newcommand\getsomebytes[3]{ % filename, start, end
  \directlua{
    local inp = assert(io.open("#1", "rb"))
    local data = inp:read("*all")
    \gdef\heretheyare{string.byte(data,#2,#3)}
  }
}

Then, in the document body, I would use:

\getsomebytes{filename.png}{4}{56} % bytes from 4 to 56 here

That would not print anything, but it would store the result in macro \heretheyare as plain text (numbers instead of letters OK, as long as the string can be parsed by LaTeX string methods, such as from package xstring). After processing, I would re-use the commands, for another file.

I wonder if it is necessary to open "all" of the file, since in some cases the file might be very large (20Mb or more) and I only need to inspect a part of it.

Answer 1

The following shows the first 5 bytes as 137 80 78 71 13

which matches the hexdump of my test png

$ hexdump.exe man2.png | head
0000000 5089 474e 0a0d 0a1a 0000 0d00 4849 5244
0000010 0000 0001 0000 5801 0208 0000 3300 b5cd
0000020 00ab 0000 7301 4752 0042 ceae e91c 0000


\documentclass{article}

\usepackage{graphicx}
\begin{document}

\includegraphics{man2.png}

\directlua{
local inp = assert(io.open("man2.png", "rb"))
local data = inp:read("*all")
local b1,b2,b3,b4,b5 = string.byte(data,1,5)
print(b1 .. ' ' .. b2 .. ' ' ..b3 .. ' ' .. b4 .. ' ' .. b5)
}
\end{document}

just change 1,5 to the start and length of the byte sequence you need

---

A version using seek (so doesn't read the whole file) and leaving the result in tex macro

\documentclass{article}

\usepackage{graphicx}
\begin{document}

\includegraphics{man2.png}

\directlua{
inp = assert(io.open("man2.png", "rb"))
}

\def\zz#1#2{%
\edef\zzhere{\directlua{
local p=inp:seek("cur",#1)
local r=inp:read(#2)
sep=""
for i,_ in string.bytes(r)
do
tex.sprint(sep)
sep=","
tex.sprint(i)
end
}}}

\zz{1}{5}

bytes 1 to 5 are :[\zzhere]
\end{document}

Answer 2

After thinking about the answer posted by David (above), I have expanded it to the following TeX code. Put here in case others with to use it.

The input file could be anything readable by TeX. The output string is plain text (comma-separated numbers from 0 to 255) so it can be parsed easily.

In this example, the input file is a PNG image. I want to know its bit depth and color type. Although there is an existing Lua library that provides the bit depth, it seems to fail for the color type. So, I get the info directly. All that is necessary to know is, where to look in the file, and what values are expected. If the desired info is not at a particular byte offset, then I can grab a sufficiently large string, parse it for some kind of header, and offset from there.

This can also look at the end of a file, which is where some file types store added info such as comments.

% compile only with lualatex
\documentclass{article}
\usepackage{mwe}
\usepackage{xifthen}
\usepackage{xparse}
% Thanks to David Carlisle, via tex.stackexchange.com:
% Output is comma-separated list of byte codes, decimal 0-255.
% Returns -1 if requested start is more than file size.
% Returns all bytes if requested number exceeds file size.
% Does not test if file exists; error if not found.
\DeclareDocumentCommand\getsomebytes { m m m } {%
  % filename, start byte (0=beginning, e=end), number of bytes
  \ifthenelse{\equal{#2}{e}}{% package xifthen
    \long\edef\herearebytes{%
      \directlua{
        inp = assert(io.open("#1", "rb"))
        local e=inp:seek("end")
        if #3>e+1 then
          inp:seek("set")
          local r=inp:read(e)
          sep=""
          for i,_ in string.bytes(r)
          do
          tex.sprint(sep)
          sep=","
          tex.sprint(i)
          end
        else
          local b=e-2-math.min(e,#3)
          local w=1+math.min(e,#3)
          inp:seek("set",b)
          local r=inp:read(w)
          sep=""
          for i,_ in string.bytes(r)
          do
          tex.sprint(sep)
          sep=","
          tex.sprint(i)
          end
        end
      }%
    }%
  }{%
    \long\edef\herearebytes{%
      \directlua{
        inp = assert(io.open("#1", "rb"))
        local e=inp:seek("end")
        if #2>e then tex.sprint(-1) else
          local w=math.min(#3,e-#2)
          inp:seek("set",#2)
          local r=inp:read(w)
           sep=""
          for i,_ in string.bytes(r)
          do
          tex.sprint(sep)
          sep=","
          tex.sprint(i)
          end
        end
      }%
    }%
  }%
} % end getsomebytes
\def\pngbitdepth#1{\getsomebytes{#1}{24}{1}\herearebytes}
\def\pngcolortype#1{\getsomebytes{#1}{25}{1}\herearebytes}
\begin{document}
This `example-image.png' is part of the `mwe' package:\par
PNG bit depth: \pngbitdepth{example-image.png} (expected 8)\par
PNG color type: \pngcolortype{example-image.png} (expected 0)\par
\getsomebytes{example-image.png}{0}{8}
First 8 bytes are: \herearebytes ~(expected 137,80,78,71,13,10,26,10)\par
\getsomebytes{example-image.png}{e}{6}
Last 7 bytes are: \herearebytes ~(expected 73,69,78,68,174,42,60,82)\par
\getsomebytes{example-image.png}{45}{3}
Bytes 45, 46, 47 are: \herearebytes ~(expected 11,252,97)\par
\getsomebytes{example-image.png}{12345}{2}
Bytes 12345 and 12346 are: \herearebytes ~(expected -1 because file is not big enough)\par
\end{document}

EDIT: Added case, if requested start byte is more than EOF, returns string -1.

EDIT2: I just put David's reply to good use, now included in version 1.52 of the "novel" package. Background:

This is specific to LuaLaTeX, and refers only to png and jpg grayscale images. Some other situations (such as color cover artwork) can be handled, but the code is different.

Problem: Some PDF files (such as PDF/X-1a) restrict the kind of images that may be included. But TeX does not detect whether or not an image meets the required specs. It is possible to call an external graphics program using shell-escape, but this creates two issues: First, the arrangement is not portable, because users on a network or online cannot rely on shell-escape and external programs. Second, processing images from within TeX can take up a lot of time, which will be repeated at each compile.

Solution: Pre-process the images using shell/batch scripts. This can be done portably, and does not use TeX. The processed images will contain a comment, declaring that they are good. Then, using the power of Lua code, the TeX document will inspect each image for the comment. This can be done quickly. If any unprocessed images are detected, a Warning will be issued. Not an Error, since it is possible that an unprocessed image is nevertheless good. Since this part does not require supporting graphics or shell-escape, it is also portable, and can be done via network or online. Also, TeX keeps track of the images, so it is not necessary to re-inspect a repeated image.

The working Lua code is based on David's answer. With all supporting LuaLaTeX code, it is too long (and too specific) to be posted here. But it can be seen in the "novel" package version 1.52, near the bottom of the "novel-Images.sty" file. The pre-processing scripts are in the documentation 'extras" folder. Have a look, if you wish.

Why this matters: LuaLaTeX can create PDF/X-1a files, which are required by some major American print-on-demand services. A program such as Adobe Acrobat Pro has built-in graphics support that can detect and correct non-compliant images. TeX does not. That is, TeX didn't have it until now!