Syntax and semantics of XDV commands (XeTeX)

Question

Donald E. Knuth has included a complete list of the syntax and semantics of DVI commands (opcode, symbolic name, number of parameters, meaning) in Computers & Typesetting Vol. E, §585, pp. 236–239. Where I can find a similar list for the additional commands in the 250–255 range, used by XeTeX?

Available information

Here is what I got (the information comes from https://github.com/simoncozens/dvipdfm-x/blob/libtexpdf/dvicodes.h):

begin_reflect 250. (No parameters???) TeX-XeT begin reflect
end_reflect 251. (No parameters???) TeX-XeT end reflect
xdv_native_font_def 252. Parameters??? Syntax???
xdv_glyphs 253. Parameters??? Possible values???
??? 254. ???
ptexdir 255. Parameters??? Possible values???

Needless to say, the information in the dvicodes.h file is lacunary. Can anyone complete this? (meaning of each command, number of parameters—if any, list of possible values)

Trying to decypher

I opened a very simple XDV file, obtained from the following TeX code:

\documentclass{article}
\usepackage{polyglossia}
\setdefaultlanguage[calendar=gregorian,hijricorrection=1,locale=mashriq]{arabic}
\setmainfont[Script=Arabic,Extension=.ttf]{Amiri-Regular}
\begin{document}
Bonjour\special{ligne x}
\begin{Arabic}يانيس\end{Arabic}
Bonjour
\end{document}

In the XDV file, some info is in specials (command xxx), that's fine. Then there is a xdv_native_font_def command followed (a) by the following bytes:

00 00 00 1E 00 0A 00 00 00 00

(b) by letter L and the absolute path to the font, (c) another four 00 bytes. Then the font 30 is called (although it has not been defined in the standard fnt_def way).

Then comes a xdv_glyphs command, followed by 2 bytes: 00 and 1F (???) and then another fnt_def command for font 50 (???) and then what looks like a lot of set_char commands for glyph 00. Since I don't know the xdv_native_font_def syntax, I probably got it all wrong.

What is really strange is that the Bonjour string of the file does never appear in the XDV file (contrarily to DVI files, where ASCII glyphs are assigned to opcodes having the numeric value of the ASCII position). Where the Bonjour string should have been, there is this byte string:

00 25 00 52 00 51 00 4d 00 52 00 58 00 55

where I can only conclude that 0025 is letter B (whose ASCII code is 0x42 and not 0x25), 0052 is letter o (whose ASCII code is 0x6F and not 0x52). Where on Earth comes the assignment B → 0x25, o → 0x52 from?

The xdv_native_font_def command appears again in the postamble. Although part of the document is in right-to-left mode, I don't see anywhere the begin_reflect and end_reflect commands, but not knowing the xdv_native_font_def syntax probably scrambles my analysis of the whole XDV file.

So any help will be greatly welcome!

For those who have no Unix tools handy, here is the hexdumped test file:

00000000  f7 07 01 83 92 c0 1c 3b  00 00 00 00 03 e8 1d 20  |.......;....... |
00000010  58 65 54 65 58 20 6f 75  74 70 75 74 20 32 30 31  |XeTeX output 201|
00000020  39 2e 30 36 2e 31 36 3a  32 31 31 30 8b 00 00 00  |9.06.16:2110....|
00000030  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000050  00 00 00 00 00 ff ff ff  ff ef 14 70 64 66 3a 70  |...........pdf:p|
00000060  61 67 65 73 69 7a 65 20  64 65 66 61 75 6c 74 a0  |agesize default.|
00000070  02 79 00 00 8d a0 fd a3  00 00 a0 02 3f 00 00 8d  |.y..........?...|
00000080  a0 fd e4 00 00 8d 92 01  2c 53 40 ef 61 70 64 66  |........,S@.apdf|
00000090  3a 64 6f 63 69 6e 66 6f  3c 3c 2f 42 49 44 49 2e  |:docinfo<</BIDI.|
000000a0  46 75 6c 6c 62 61 6e 6e  65 72 28 54 68 69 73 20  |Fullbanner(This |
000000b0  69 73 20 74 68 65 20 62  69 64 69 20 70 61 63 6b  |is the bidi pack|
000000c0  61 67 65 2c 20 56 65 72  73 69 6f 6e 20 33 35 2e  |age, Version 35.|
000000d0  37 2c 20 52 65 6c 65 61  73 65 64 20 4d 61 72 63  |7, Released Marc|
000000e0  68 20 33 2c 20 32 30 31  39 2e 20 29 3e 3e 96 02  |h 3, 2019. )>>..|
000000f0  ee 00 fc 00 00 00 1e 00  0a 00 00 00 00 13 2e 2f  |.............../|
00000100  41 6d 69 72 69 2d 52 65  67 75 6c 61 72 2e 74 74  |Amiri-Regular.tt|
00000110  66 00 00 00 00 c9 fd 00  1f dd 80 00 07 00 00 00  |f...............|
00000120  00 00 00 00 00 00 05 d2  00 00 00 00 00 00 0a 98  |................|
00000130  80 00 00 00 00 00 0f ca  80 00 00 00 00 00 12 20  |............... |
00000140  00 00 00 00 00 00 17 18  80 00 00 00 00 00 1c 20  |............... |
00000150  00 00 00 00 00 00 25 00  52 00 51 00 4d 00 52 00  |......%.R.Q.M.R.|
00000160  58 00 55 93 fd 00 13 27  c0 00 05 00 00 00 00 00  |X.U....'........|
00000170  00 00 00 00 0a ca 80 00  00 00 00 00 0d 3b 80 00  |.............;..|
00000180  00 00 00 00 0e f4 c0 00  00 00 00 00 11 40 40 00  |.............@@.|
00000190  00 00 00 06 5e 0e 9b 0e  7c 04 ef 05 2d 93 ef 07  |....^...|...-...|
000001a0  6c 69 67 6e 65 20 78 fd  00 1f dd 80 00 07 00 00  |ligne x.........|
000001b0  00 00 00 00 00 00 00 05  d2 00 00 00 00 00 00 0a  |................|
000001c0  98 80 00 00 00 00 00 0f  ca 80 00 00 00 00 00 12  |................|
000001d0  20 00 00 00 00 00 00 17  18 80 00 00 00 00 00 1c  | ...............|
000001e0  20 00 00 00 00 00 00 25  00 52 00 51 00 4d 00 52  | ......%.R.Q.M.R|
000001f0  00 58 00 55 8e 8e 9f 1e  00 00 8d 92 00 e7 92 00  |.X.U............|
00000200  fd 00 05 dc 00 00 01 00  00 00 00 00 00 00 00 01  |................|
00000210  be 8e 8e 8c f8 00 00 00  2c 01 83 92 c0 1c 3b 00  |........,.....;.|
00000220  00 00 00 03 e8 02 79 f2  7f 01 97 00 00 00 03 00  |......y.........|
00000230  01 fc 00 00 00 1e 00 0a  00 00 00 00 13 2e 2f 41  |............../A|
00000240  6d 69 72 69 2d 52 65 67  75 6c 61 72 2e 74 74 66  |miri-Regular.ttf|
00000250  00 00 00 00 f9 00 00 02  14 07 df df df df df df  |................|
00000260

Apart from dvipdfmx, I'd recommend looking at dvisvgm — it handles xdv files. — ShreevatsaR, Jun 16, 2019 at 19:44
(BTW, presumably you mean Volume B and not Volume E; also the repository you linked seems to be an old mirror: XDV version has changed and it may not be perfectly relevant to the file above.) — ShreevatsaR, Jun 16, 2019 at 19:52

yannis · Accepted Answer · 2019-07-04 04:04:58Z

Based on the dvisvgm sources and dvipdfm-x sources: The truly XDV-specific opcodes are (as of the current Version 7) only three:

252 (fc): This is to define a font (code refers to it as XDV_NATIVE_FONT_DEF or XFontDef), and is the most complicated of the three. Parameters are:

fontnum[4] ptsize[4] flags[2] psname_len[1] fontname[psname_len] fontIndex[4]

followed by up to (2 + 4 * 65535) more bytes depending on the flags.
253 (fd). This is a “string of glyph IDs with X and Y positions”, referred to in code by XDV_GLYPHS or XGlyphArray. Parameters are:

w[4] n[2] xy[(4+4)n] g[2n]

where w is the total width of the glyph array, n is the number of glyphs, xy is a sequence of (dx, dy) pairs (the relative horizontal and vertical positions of each glyph), and g contains the “FreeType indices of the glyphs to typeset”.
254 (fe): This is similar except it includes “a leading array of UTF-16 characters that specify the "actual text" represented by the glyphs to be printed. It usually contains the text with special characters (like ligatures) expanded so that it can be used for text search, plain text copy & paste etc. This XDV command was introduced with XeTeX 0.99995 and can be triggered by \XeTeXgenerateactualtext1”. So its parameters are:

parameters: l[2] t[2l] w[4] n[2] xy[8n] g[2n]

I don't think the TeX-XeT commands 250–251 nor the pTeX command 255 are used by XeTeX, which is consistent with you not seeing them in the file.

The hexdump in the question starts with f7 = 247, the DVI “pre” command, and the next byte is the DVI version, which here is 07. So we're looking at (XDV) version 7, as expected.

So in your file when you see (at either byte offset 278 or 423) bytes like fd 00 1f dd 80 00 07 00 00 00 and so on, it's actually not just two bytes that are the parameters, but rather 00 1f dd 80 are w, then 00 07 are n (the number of glyphs), then the next 56 bytes are xy or (dx, dy) (the offsets for each of these 7 glyphs), then the next 14 bytes are g or glyphs (the 7 glyphs). Needless to say, these 7 in your example are Bonjour:

    00 25 00 52 00 51 00 4d 00 52 00 58 00 55

As you observe, these are not ASCII codes, so where does this mapping of 00 25 to B, etc come from? Well it's the same as with the regular DVI format: these are the positions of the glyphs in the font, and the font can choose to put any glyph at any position. This is confirmed by opening the font and counting positions: maybe FontForge can show it but I couldn't find it in the UI, but I could find it with fonttools:

$ ttx amiri-regular.ttf 
Dumping "amiri-regular.ttf" to "amiri-regular.ttx"...

and the file contains:

 <GlyphID id="37" name="B"/>

where 37 is 0x25, etc.

In FontForge you need to go to Encoding -> Rencode -> Glyph Order, then the numbers in the status bar will show the glyph ids. — خالد حسني, Jun 17, 2019 at 1:24
So you are saying that there is no information about change of direction in the XDV file, be it right2left or horizontal2vertical? — yannis, Jun 17, 2019 at 6:41
@yannis Honestly, like most of my answers, I had no clue when I read the question and the answer is the total extent of whatever understanding I gleaned since. :-) See the answer of Khaled Hosny; he has worked on XeTeX and was at least for a while the main person doing so; I know very little about it. But having said that, yes what I gather from looking at xetex.web/xetex.pdf is that indeed the XDV file contains nothing about change of direction. See e.g. section 652 and 1508: “RTL text is reversed… by the ship_out routine and is written […] without any begin_reflect or end_reflect commands”. — ShreevatsaR, Jun 17, 2019 at 7:05
@yannis: XeTeX uses eTeX's TeXXeT which reorders the TeX nodes internally and writes the output in visual order (and it reorders everything, even special nodes that should not be reordered). A few years ago I tried to replace that with the original TeX--XeT that has these reflect opcodes, but it turned out it broke some math layout in unexpected ways and had to be reverted. — خالد حسني, Jun 18, 2019 at 8:52

David Carlisle · Accepted Answer · 2019-06-16 20:29:51Z

Not a lot better than the source you gave but the xetex output is read by xdvipdfmx and the source for that in texlive svn has dvicodes.h which has

                    /* XeTeX ".xdv" codes */
#define XDV_NATIVE_FONT_DEF 252 /* fontdef for native platform font */
#define XDV_GLYPHS          253 /* string of glyph IDs with X and Y positions */
#define XDV_TEXT_AND_GLYPHS 254 /* like XDV_GLYPHS plus original Unicode text */

#define PTEXDIR             255 /* Ascii pTeX DIR command */

These are handled by dvi.c in the same directory, but I guess your C is better than mine:-)

There are some comments as to the expected byte layout in the C eg

case XDV_GLYPHS:
  need_XeTeX(opcode);
  get_and_buffer_bytes(fp, 4);            /* width */
  len = get_and_buffer_unsigned_pair(fp); /* glyph count */
  get_and_buffer_bytes(fp, len * 10);     /* 2 bytes ID + 8 bytes x,y-location per glyph */
  break;
case XDV_TEXT_AND_GLYPHS:
  need_XeTeX(opcode);
  len = get_and_buffer_unsigned_pair(fp); /* utf16 code unit count */
  get_and_buffer_bytes(fp, len * 2);      /* 2 bytes per code unit */
  get_and_buffer_bytes(fp, 4);            /* width */
  len = get_and_buffer_unsigned_pair(fp); /* glyph count */
  get_and_buffer_bytes(fp, len * 10);     /* 2 bytes ID + 8 bytes x,y-location per glyph */
  break;
case XDV_NATIVE_FONT_DEF:
  need_XeTeX(opcode);
  do_native_font_def(get_signed_quad(dvi_file));
  break;

dviasm (despite its name) can show xdv files, I had to change the font loading to find the font, but on your test file it reports

[preamble]
id: 7
numerator: 25400000
denominator: 473628672
magnification: 1000
comment: ' XeTeX output 2019.06.16:2127'

[postamble]
maxv: 633.947250pt
maxh: 407pt
maxs: 3
pages: 1

[font definitions]
fntdef: "/usr/local/texlive/2018/texmf-dist/fonts/truetype/public/amiri/amiri-regular.ttf" at 10pt

[page 1 0 0 0 0 0 0 0 0 0]
xxx: 'pdf:pagesize default'
down: 633pt
push:
  down: -605pt
  down: 575pt
  push:
    down: -540pt
    push:
      right: 300.325195pt
      xxx: 'pdf:docinfo<</BIDI.Fullbanner(This is the bidi package, Version 35.8, Released May 1, 2019. )>>'
      w: 2.929688pt
      fnt: "/usr/local/texlive/2018/texmf-dist/fonts/truetype/public/amiri/amiri-regular.ttf" at 10pt
      setglyphs: 31.865234pt gid37(0pt) gid82(5.820312pt) gid81(10.595703pt) gid77(15.791016pt) gid82(18.125000pt) gid88(23.095703pt) gid85(28.125000pt)
      w0:
      setglyphs: 19.155273pt gid2388(0pt) gid4497(10.791016pt) gid4466(13.232422pt) gid2021(14.956055pt) gid2083(17.250977pt)
      w0:
      xxx: 'ligne x'
      setglyphs: 31.865234pt gid37(0pt) gid82(5.820312pt) gid81(10.595703pt) gid77(15.791016pt) gid82(18.125000pt) gid88(23.095703pt) gid85(28.125000pt)
    pop:
  pop:
  down: 30pt
  push:
    right: 231.570312pt
    setglyphs: 5.859375pt gid447(0pt)
  pop:
pop:

With the tex file you gave apart from changing the font line to

\setmainfont[Script=Arabic]{Amiri}

ShreevatsaR · Accepted Answer · 2019-06-17 01:58:49Z

8

The definitive source is the XeTeX source tree, and specifically the xetex.web file. Quoting from it:

\yskip\noindent Commands 250--255 are undefined in normal .{DVI} files, but the following commands are used in .{XDV} files.

\yskip\hang\vbox{\halign{#&#\hfil\cr |define_native_font| 252 & |k[4]| |s[4]| |flags[2]| |l1| |n[l]| |i[4]|\cr & |if (flags and COLORED) then| |rgba[4]|\cr & |if (flags and EXTEND) then| |extend[4]|\cr & |if (flags and SLANT) then| |slant[4]|\cr & |if (flags and EMBOLDEN) then| |embolden[4]|\cr }}

\yskip\hang|set_glyphs| 253 |w[4]| |k[2]| |xy[8k]| |g[2k]|.

\yskip\hang|set_text_and_glyphs| 254 |l[2]| |t[2l]| |w[4]| |k[2]| |xy[8k]| |g[2k]|.

\yskip\noindent Commands 250 and 255 are undefined in normal .{XDV} files.

edited Jun 17, 2019 at 1:58

ShreevatsaR

43.9k10 gold badges112 silver badges144 bronze badges

answered Jun 17, 2019 at 1:14

خالد حسني

22.8k1 gold badge81 silver badges118 bronze badges

Thanks, this would be definitive. I added an image of the typeset version (from running weave on the web file) if you don't mind; feel free to revert. (Needed some tweaks to get it to weave and typeset; looks like no one uses that anymore :))
– ShreevatsaR
Jun 17, 2019 at 1:58
1

@ShreevatsaR: was about to do this but my tex installation was broken and took me a bit to fix it :)
– خالد حسني
Jun 17, 2019 at 2:15
I've collected all the tweaks needed to run weave on xetex.web and created a merge request here… not sure what the next steps are to get it looked at.
– ShreevatsaR
Aug 7, 2019 at 3:14
@ShreevatsaR: no idea either.
– خالد حسني
Aug 8, 2019 at 11:07
@KhaledHosny how could we contact with you?
– Mamoun Mohammed
Nov 6, 2020 at 21:06

Add a comment |

Stack Exchange Network

Syntax and semantics of XDV commands (XeTeX)

Available information

Trying to decypher

3 Answers 3

Your Answer

Not the answer you're looking for? Browse other questions tagged
xetex
dvi
or ask your own question.

Hot Network Questions

Syntax and semantics of XDV commands (XeTeX)

Available information

Trying to decypher

3 Answers 3

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged xetexdvi or ask your own question.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
xetex
dvi
or ask your own question.