# How to sequentially “feed” a LaTeX document from external data source?

I have a quite complex C program that has the ability to generat raw text output to text files that are then printed on a matrix printer. These lists are now meant to receive a typographical update to finally exist as a PDF that can be printed on a regular laser printer. For this I intend to use LaTeX. I can not exchange the C program, only modify the "printing/text output" functionality.

The general layout of the document to be generated is well-defined, but the number of rows of data is not.

So, in a first step I wrote a LaTeX "template" file, like this:

\documentclass{article}
\begin{document}
Dear --MRMRS-- --NAME--, we hope ...

And let my program replace (in the end with a sed 's/KEYWORD/programoutput/' command) the keywords --MRMRS--, --NAME-- by the data it produces in raw form. This works pretty well, to this point. As already mentioned, the length of some parts of the generated files is not clear beforehand. For example, later in the document there will be tables with a well-defined structure, but these tables can be different in length each time, so I can not simply define a given Number of --ROWxCOLy-- beforehand:

\begin{tabular}{cccc}
Col1 & Col2 & Col3 & Col4 \\
\hrule
% now how to fill the content sequentially without knowing the size beforehand?
\end{tabular}

Thank your, we hope to hear from you on --DATE-- ...
\end{document}

The data that comes looks currently like

Col1 Col2 Col3 Col4
Col1 Col2 Col3 Col4
Col1 Col2 Col3 Col4
COLSPANNED     LINE
Col1 Col2 Col3 Col4
Col1 Col2 Col3 Col4
... variable amount of lines

There can be special lines that need a full column span (span whole width)
but they come in well-defined order

The first solution that comes to mind is to hardcode LateX code directly in the data generating program, to have one keyword for the whole table, that is then generated with hardcoded LaTeX code in the program. But I'd like to avoid this as much as possible for obvious maintainance reasons (keep layout and logic separated as much as possible).

What other possible solutions are there to feed a LaTeX document with well-defined, but unknown length data?

• Can you give a complete example of the expected output? – Alexey Jun 26 '15 at 9:30
• @Alexey Currently the program produces space-separated raw text columns with a variable number of rows. I need to get this into LaTeX code without hardcoding LaTeX commands into my C program source. Updated question with a basic example. – Foo Bar Jun 26 '15 at 9:37
• I see, your question about how to wrap you output in LaTeX, i might have misunderstood it. I think that there is no choice other than process each output line individually and convert it to LaTeX, possibly wrapping in some LaTeX commands. – Alexey Jun 26 '15 at 9:43
• I mean, is your problem any different from the problem of wrapping/converting/sanitizing plain text into HTML? I think there should exist some solutions for the latter. – Alexey Jun 26 '15 at 9:44
• I would have used a solution similar to the one for HTML generation: i would have used some template engine, like eRuby. – Alexey Jun 26 '15 at 9:48

# Automatic processing of data for TeX / LaTeX

The problem in your question is the term "sequentially". It is impossible to generate one TeX/LaTeX document continuously. A TeX / LaTeX document that produces valid output has a begin and more important, an end. When the document is finished without errors, the output is complete and the TeX / LaTeX job is done. You cannot "feed" data into that document, once it is done.

What you can do is to build a LaTeX frame document, \input some_external_data and re-produce the output every time the external data changes.

The topic is: At the time the LaTeX job runs, the data you want to produce has to be defined in content and length, as the output document wont update automatically later, when the data changes, without running the LaTeX job again. At least not without quite complex, customized methods, that greatly depend on your used viewer or output medium.

### To improve your original approach

It might help to change the way you process the data

Dear --MRMRS-- --NAME--, we hope ...

Here you use a self defined template language to produce a TeX file. I think that is, what you mean by "hard coded".

As its a good practice to separate output form (layout) and logic, it is also a good practice to separate generated data from the template as long as possible and to translate the set of information (the input or the data) in a way, the next processor understands (next processor is LaTeX in our case).

I show what I mean: LaTeX does not know what to do with --MRMRS-- and such constructs, thought the TeX machine is generally able to setup such a parser. But that would make things quite complex and hard to control and debug. So keep in the language LaTeX language domain, when you define your template:

Dear \MRMRS{} \NAME{}, we hope ...

Let's keep it simple and say that is our whole pattern, then the data set in text form might be

Mrs
Moneypenny

The C program might translate this input data into a form, that is known by the C language:

struct greeting {
char* mrmrs;
char* name;
};

Now the purpose of the C program (or whatever) is to translate that into the LaTeX language

\def\MRMRS{Mr}
\def\NAME{Moneypenny}

You can now read in the processed data into the LaTeX program, that is a true LaTeX program, not a template to-be-processed by whatever and every step from the raw data to the output document can be debugged separated from the other processes.

• The LaTeX compiler will run after the "feeding" is finished in my case. I feed the .tex file sequentially and after this is done, I run pdflatex on the complete .tex file. The question is how to repeat / feed LaTeX commands without hardcoding them outside of the .tex file? I think about using (La)TeX own control mechanisms somehow (loops, if-else), I'm just not sure on how to do this efficiently. – Foo Bar Jun 26 '15 at 9:54
• Outside of what TeX file? You need to keep the TeX commands somewhere, and mix them with plain text from elsewhere. IMO this is called a template. – Alexey Jun 26 '15 at 9:56
• @Alexey I have a termplate tex file, that has keywords in it. My C program does this: copy the template to a new name, subsitute the keywords with real data, and finally run pdflatex. The next time the C program runs, it does the same (copy template to working copy, replace Keywords with data, run pdflatex). Now, the problem is that some tables are not fixed-length, so I can not define 10 keywords, I need to define X keywords (variable). This is obviusoly not possible without using control structures somewhere in the LaTeX document. – Foo Bar Jun 26 '15 at 9:59
• I see, but i do not understand why you would not like to try an existing template engine instead of inventing one with keyword substitution. I suggested eRuby because it is made to generate plain text file, it is not specific to HTML, unlike some others. – Alexey Jun 26 '15 at 10:02
• @FooBar please give us a concrete example, that we can figure the set of objects / data, you want to loop over or the flags you want to branch at. But if I look at your data set in your question, you might be able to write a macro that loops over the tokens of your data and produce TeX tokens from the data. But in general, TeX isn't that good in formatting any data. Many characters have special meaning, that have to be escaped or otherwise replaced, which is possible. But your idea to pre-format the data with a C program is not the worst, in my opinion. – ikrabbe Jun 26 '15 at 10:02

To "keep layout and logic separated as much as possible", a cleaner approach would be to have your program write XML instead of raw text.

This can then be transformed (probably using XSL) to LaTeX or any other output format you like, including raw text.

Of course, this takes additional time.

For a quick and dirty solution: If the column text is fixed width, you might consider using tabbing instead of tabular. That way, you just define the maximum numbers of columns, but you use only those you need. So there is no need to count the columns.

An advanced example: setting tabs by declaration rather than by example

• Some people, me too, think of XML as a failure by design: harmful.cat-v.org/software/xml. But principally the pre-processing outside of TeX is a useful approach. I edit my answer to reflect this purpose. – ikrabbe Jun 26 '15 at 11:53