# Data structures for TeX

I often find myself wanting to use standard data structures in TeX. For example, lists or maps. Each time this need comes up, I reinvent the wheel. For some reason, I find myself using stacks fairly frequently.

Is there a standard package (either plain TeX or LaTeX) that offers a convenient interface to data structures? They would be most useful if the structures could be combined so that you could have lists of maps, for example.

-
Generate your TeX from the language of your choice giving you all the facilities of whatever language you choose and letting you leverage off your existing knowledge of that language. –  user1189687 Jul 31 '12 at 11:26

One of the ideas of expl3 is to provide a number of standard programming tools, and data structures are included in this. Currently, the structured data types available are sequences, stacks and property lists. Sequences and stacks have the same underlying structure: they are different ways of looking at the same thing. So we might have

\seq_new:N \l_my_seq
\seq_put_right:Nn \l_my_seq { Some-tokens } % Sequence-like access
\seq_push:Nn \l_my_seq { More-tokens } % Stack-like access, puts to left


Property lists are a key-value type data type

\prop_new:N \l_my_prop
\prop_put:Nnn \l_my_prop {  key-name } { value }


Property lists are 'unordered', whereas sequences/stacks are ordered. In both cases there are functions available to map to each item in the stored data, for example \seq_map_function:NN.

New data types can be added to expl3 as needed: this depends on the requirements of the LaTeX3 Project but also of others. (Note of course that I'm on the LaTeX3 Project!)

An alternative set of packages providing some similar ideas to expl3 including a number of data-related structures is the combination of etoolbox and etextools. These two packages provide functions for mapping to lists and so on.

You might also look at the datatool package, which can be used to construct tables in LaTeX and access the data in a database-like fashion.

One other note. LuaTeX provides access to the native Lua data storage methods, which if you are doing a LuaTeX-only project would of course be the first place to look.

-
I've never looked at expl3, the naming has been off-putting. I take it it makes _ have catcode 11, same with :? Or I guess it could \def\seq_#1:{...}. At any rate, I'll take a look, thanks. –  TH. Aug 29 '10 at 9:12
You get used to the naming :) Yep, inside \ExplSyntaxOn...\ExplSyntaxOff colon and underscore have catcode 11. –  Will Robertson Aug 29 '10 at 10:17
As Will says, the naming uses different 'letters' to traditional TeX. It's no more or less logical than using @ as a letter, really, and fits more with what a lot of other languages do. (Plus avoids the business where @ is used in a more-or-less random way in names.) –  Joseph Wright Aug 29 '10 at 10:44
I do dislike how m@cros in LaTeX st@rt @cquiring m@re @nd m@r@ '@'s. And then, at the end of some words@, there are@ one@@ or@@ more@@@ '@'s. –  TH. Aug 29 '10 at 11:14
Go to a good programmer who has not seen any TeX code before and show her things like \prop_put:Nnn and ask what this does. If she can answer it without asking back, the language (API) is good. If she can't, then rethink the language. Source code must be readable without having to translate everything in your head. Why? The more complex the translation process is, the more expensive is the code to maintain. And the less likely it is that someone actually uses that language. -- So my advise is to drop as much of the input language as you can and switch something more sensible, such as Lua.:) –  topskip Aug 29 '10 at 13:50

I recently discovered the package arrayjobx (nee arrayjob) which has transformed my TeX usage. Linked lists and indexed references are very simple to do. Arrays combined with fp have allowed me to manage numerical data in TeX for display. Reading the code for arrayjobx was also very enlightening.

\documentclass{minimal}
\usepackage{arrayjobx}
\makeatletter
\usepackage{trimspaces}
\def\trimspace#1{\trim@spaces@in{#1}}
\makeatother

\begin{document}
\newarray\MyData

Years       & 1996 & 1997 & 1998 & 1999 & 2000 &%
Revenues    & 1034 & 1226 & 1445 & 1503 & 1992 &%
Costs       & 668  & 706  & 813  & 946  & 1010  }
\dataheight=6

\def\returnYear(#1){\checkMyData(1,#1)\trimspace\cachedata \cachedata}
\def\returnData(#1){\expandafter\checkMyData(#1)\trimspace\cachedata \cachedata}

\def\mycolumn{3}
\def\myrow{2}

Our expected \returnHeading(\myrow) in \returnYear(\mycolumn) were
\\$\returnData(\myrow,\mycolumn).

\end{document}


On the other hand, I am not familiar with the packages mentioned in other answers. Maybe they render arrayjob obsolete.

-
I remember briefly looking through this package earlier this year. It seems pretty nice; expl3 doesn't have any multi-dimensional data structures by default (yet?). –  Will Robertson Sep 21 '10 at 2:25
Brandon there is an update. It is a very good package, but use arrayjobx, which is the latest version. Previous one has conflicts with amsmaths –  Yiannis Lazarides Sep 21 '10 at 10:25

Did no one mention the bridges to Lua in Luatex? Well, Patrick did, oblliquely.

1. Luatex itself provides a number of low-level primitives to the Lua half of the implementation, most importantly the \directlua command. Using this level is a bit fiddly.
2. Context 4 has, unsurprisingly, nice support for Lua. For example, \startluacode ... \endluacode allows Lua to be naturally embedded in Context documents. This is important, because Context uses Lua to manage the whole compilation process.
3. There's high-level Lua infrastructure for Latex appearing, codename Lualatex, with the obligatory \LuaLaTeX macro, and similarly nice environments for embedding code. There's not much in Texlive 2009, and I've not made the jump to 2010 yet, but the document about supporting Luatex's hook in Latex, which Patrick pointed out to me, looks appetising.
4. There's some Luatex support for Plain Tex, codename Luatextras.

Lua has a nice, efficient, minimalist approach to datastructures, based on its notion of tables, that fuses lists and associative arrays, and is used together with its novel approach to lightweight closures as its basis for a prototype-based object system.

It's good, take a look.

-
I'm fairly familiar with Lua and I don't really like its "everything is a table" approach and setting values in meta tables to control behavior is not a lot of fun. It's also anything but efficient. Maybe in comparison to TeX... –  TH. Sep 20 '10 at 17:48
@TH. can you elaborate on "It's also anything but efficient"? What efficiency do you mean? –  topskip Sep 20 '10 at 19:04
@Patrick: I mean that it is slow. Very slow. Maybe LuaTeX uses a JIT so it's faster, I'm not sure. –  TH. Sep 20 '10 at 19:24
Either my memory is faulty (very possible) or Lua has gotten a lot faster. I just looked at some of the not-terribly-meaningful-but-we-all-use-them-anyway benchmarks that compare programming languages, and for a scripting language, Lua isn't bad. Looks like I need to reevaluate. (I still dislike meta tables though.) –  TH. Sep 20 '10 at 19:47
@TH.: Luatex uses the interpreter, not Lua JIT. The efficiency I meant is overall: it's at the fast end of dynamically-typed interpreters and it's very space efficient, which is more important, I think, in the context of Luatex. Cf. mail-archive.com/dev-luatex@ntg.nl/msg00725.html. Tex is efficient, very efficient for a macro-expansion language. –  Charles Stewart Sep 21 '10 at 12:52

It seems that etoolbox contains several sorts of list processing which is fantastic as I've written similar code over and over.

-

I remember seeing some time ago in this site a link to lambda-lists aka lists in TeX's mouth by Alan Jeffrey. I haven't yet tried it out, though, but it sure seemed nice and functional.

-