35

I am in the early stages of trying to replace an existing document generation system based on Microsoft Office apps (Word and Excel).

The system runs once a year to generate over 300,000 individual documents of three different types (the new system may be an "on demand" for an individual document instead of batch processing all at once). Two of the types are mostly standard text with value place-holders, and some conditional logic for inclusion/exclusion of certain elements (standard paragraphs and/or values). One of the types includes a bar chart based on individual calculations. All of the values for replacement and conditional logic are sourced from an ASCII file, but some of the values are calculated from the data.

The current system is very slow and error prone (at run time) and requires a complex system of machines, threads and message queues to scale the processing resources to a level that will get the job done within two weeks or so. Basically, there are three Word Document templates that include the value place-holders and conditional logic and text. The templates are processed with Office interop libraries to create an instance document. In the case of one of the types, Excel is used to create a bar chart that is injected (OLE embedded) into the instance document. The Word instance documents are then converted to (saved as) PDF.

I only know a little about TeX (brushed up against it during my many years of Emacs use), but it seems that it might serve as a good basis to replace the behemoth described above. The problem is that I need some guidance as to whether or not TeX would be a good route to persue (performance being a key factor), and some pointers to resources that can accomplish the more obscure needed tasks (I know PDF generation is no issue).

The final system would execute on Windows machine(s), and programmatic processing would be done with .NET or Java most likely.

Improve this question
12
  • 2
    Review this similar question tex.stackexchange.com/questions/3697/…
    – R. Schumacher
    Jul 31, 2012 at 20:27
  • 1
    Is it a single very large ASCII file for all the data? How big? How many pages will your typical final text document (types 1 and 2 if I understand correctly) contain? What kind of computations are needed to produce the extra data? Lots of floating point computations? How complicated are the bar charts? Only one per type 3 document? Lots of data points to produce each chart?
    – Bruno Le Floch
    Jul 31, 2012 at 21:08
  • 5
    What is it that the definition of done, i.e. when would you accept that a TeX solution is better than your current system? Does "performance being a key factor" mean that you want it faster? 300000 docs / 14 days / 24h / 60min = 15 docs / min if I am not mistaken. Depending on the size of your documents (or the number of data points for bar plots), this might be a challenge for TeX as well. At least assuming that the parallel potential for a TeX-based solution is the same as for your current one(?).
    – Christian Feuersänger
    Jul 31, 2012 at 21:28
  • 1
    TeX is certainly a good tool for the automatic typesetting part. If your logic/processing part in .NET or Java use a template engine and outputs a tex-file this sounds like a good solution. I would hesitate to recommend to use TeX for a lot of processing, but there are now even tools to create nice plots from ASCII data (e.g. pgfplots).
    – Alexander
    Jul 31, 2012 at 23:17
  • 1
    As @Alexander hints to, TeX is very good for the typesetting part, but is a bit slow for e.g., floating point computations. Comparing dates, or amounts, is fast, but you will need a bit of custom code to optimize that part: parsing arbitrary dates is obviously slower than parsing the more strictly defined records which you probably have. All in all, I think the task is doable in TeX, but it would be helpful if you can provide us with more specific example files (not necessarily with the correct data, if it is sensitive), and desired output.
    – Bruno Le Floch
    Aug 1, 2012 at 7:29

1 Answer 1

Reset to default
14

First of all, I can only chime in with commentors to say that TeX is certainly one of the best systems you can find for this kind of task.

As your question is not very specific and some pointers have already been given, let me just give some examples for comparable uses and suggestions for proceeding further. Feel free to ask more specific questions ;-)

One example of a (commercial) data driven document generation system implemented entirely in TeX is my DocScape. You can find some references here; I also gave some examples in this answer.

To give you some numbers on performance: A german federal government ("Niedersachsen") is using a TeX-based system for publishing budget documents (budget plan, reports and a lot of other stuff). About 16,000 people from the state administration are involved, maintaining data. Every one of them can generate a preview (of about 2-10 pages) at any time, which leads to up to 300 documents generated in parallel at any time (on an AIX mainframe).

Once a year, several volumes of 1000+ pages are generated for print, plus a couple of intermediate versions.

See for instance the last budget report.

In general, I'm probably not the right person to comment on performance, because DocScape is sadly rather inefficient from a macro programming point of view, so I can't really report any speed records for my own projects.

TeX itself, on the other hand, certainly is a role model of efficiency, because it hasn't been affected by software bloat for at least the last 30 vears. So you shouldn't run into any performance problems at all. In particular if you're generating a lot of independent documents, you can run as many TeX processes in parallel as you have processors in your machine, further speeding up things.

Here some further hints on how to proceed.

  • First, I would preprocess the input (ASCII) data, transforming either into XML or some "pseudo" TeX data notation.
  • This doesn't mean you need to generate complete TeX documents, but at least inserting some control sequences to markup document and data structures, images etc. will make later processing with TeX much easier.
  • I would by all means do all the number crunching during preprocessing, in particular for the bar charts. The charts themselves can then be plotted with TikZ.
Improve this answer
4
  • Thank you very much for this answer - the information is helpful.
    – Edwin McConnel
    Aug 6, 2012 at 16:32
  • 2
    I have been making slow progress so far, but the experiments have been somewhat good. I've recently gotten diverted in trying to convert the existing Word templates to LaTeX. I finally got ok results using writer2latex (after converting to .odt), but it does not bring over the large font text boxes at top of the front page (Garamond size 60). I've seen from example documents that such large font headers are possible; I just need to figure out how to replicate it. Other than that, using a template engine producing TeX files from a "TeX"-ish base template seems to be the way to go.
    – Edwin McConnel
    Aug 6, 2012 at 16:40
  • "such large font headers are possible" should read "large font boxes".
    – Edwin McConnel
    Aug 6, 2012 at 17:10
  • @EdwinMcConnel If you have specific questions in this regard, please put them as separate questions on this site.
    – Stephan Lehmke
    Aug 6, 2012 at 17:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .