Using \fontdimen as an array to store data

Reading through `pi.tex`, I discovered the trick of using `\fontdimen` as an array of dimensions (which is equivalent to storing integers between ±(2^31-1)sp). Namely,

``````\font\x=cmr10 at 1sp % or whatever font that is not used in the document.
\fontdimen 10000 \x = 1pt
\showthe\fontdimen 10000 \x % => 1pt
\bye
``````

This method of storage appears to be very efficient memory-wise (in TeXLive 2012, pdfTeX and XeTeX are happy to store a bit less than 3000000 such parameters, LuaTeX extends memory as needed), and speed-wise (I benchmarked access and modification to be just as fast as for usual dimen registers, once the space has been allocated).

The main drawback I see is that the size of the array must be declared as soon as the new font is introduced (or rather, before another font is defined):

``````\font\x=cmr10 at 1sp
\font\y=cmr10 at 2sp
\fontdimen1000\x=1pt % => error: '\x only has 7 fontdimen parameters'
\bye
``````

Are there any other drawbacks to using font memory for the storage of large amounts of data? Is it frequent for a document to reach the bounds of […]TeX's font memory (in which case I shouldn't use it for other purposes than fonts)? If not, I am thinking of using this technique for `l3regex`, and perhaps to store some tables of values for `l3fp`.

-
The TeXbook puts no limits on how many additional font parameters can be created, other than as you've noted requiring that they are created as soon as the font is loaded. The obvious downsides are that these are global assignments (so good for constants but not so great for other things), and that you need to avoid any 'real' font or pick high-numbered parameters so you are 'safe'. But you already know all of this ... –  Joseph Wright Jan 28 '13 at 8:39
@JosephWright I think it is useful to collect all downsides in a single answer (including the few that I mention, and the fact that assignments are global). Actually, I am wondering what the high-numbered `\fontdimen` parameters are used for, except for that trick. Should I ask a separate question? Also, there may be a risk that the values of those parameters are included in the resulting pdf? –  Bruno Le Floch Jan 28 '13 at 15:50
Knuth left the font dimensions flexible: after all, you need more for a math font than a text font, and LuaTeX/XeTeX are using this to backport the additions Microsoft have made to this area. I guess Knuth envisaged more developments/specialism in the same way he imagined people might create new types of whatsit, which did not really happen. –  Joseph Wright Jan 28 '13 at 16:18
Reading through `tex.web` I see another potential pitfall: the extra fontdimen parameters are included in the format. For l3regex I would be using the fontdimen parameters as (global) scratch space, which means that l3regex should not be used to help setting up the kernel (not much of a problem, I guess). –  Bruno Le Floch Jan 30 '13 at 8:53
Not an issue now, but you can imagine that we might put `l3regex` into the kernel at some point simply to avoid the 'now load X' business you have in LaTeX2e: easier if all of these things are 'out of the box'. That said, I'm not clear on how we might div up 'thinks in a format file' and 'thinks loaded after the format file'. –  Joseph Wright Jan 30 '13 at 8:55

1 Answer

It's possible to do this, the main disadvantage of course is they are global. Having loaded the font you can use other parts of its structure as well. This old TUGBoat paper of Jonathan Fine's discusses using the ligature tables of such fake fonts to encode state machines.

-
This article is very interesting, thank you. A couple follow-up questions: is it possible to edit the ligature table from within TeX (I think not, except perhaps in LuaTeX), or `\write` a tfm file for TeX to read back? –  Bruno Le Floch Feb 3 '13 at 17:20
no to both: well for the latter you could write the ascii pl version of the tfm and then \write18 a call to pltotf to make a tfm file. –  David Carlisle Feb 3 '13 at 17:44
The main question if we want to write tfm files is to know if a valid tfm file can end with a trailing new line. Can it? Building the tfm file could reasonably be done from TeX, I guess, since we would only use a very restricted set of features. –  Bruno Le Floch Feb 5 '13 at 8:29
the tfm itself is binary you couldn't write that (could you?) but the pl file can have trailing white space. –  David Carlisle Feb 5 '13 at 12:19
I'd rather avoid the need for shell escape, so writing a pl file would not be good enough. You can write binary data with `pdftex -8bit` and with `luatex`, not with `xetex` (since that engine can only write valid utf8). See tex.stackexchange.com/questions/8729 . Would another font file format be "less binary"? The `pfb` I can find in my system seem to have some non-binary parts. –  Bruno Le Floch Feb 5 '13 at 12:44