How would it be possible to automatically load installed(!) packages during compile time?

This is more of an hypothetical question. It is NOT about to install and load missing packages which are declared but not installed, as done for example by MiKTeX.

Let's say one is typing a report or an article. While typing the code, e.g.

\documentclass[a4paper,12pt,onecolumn]{scrartcl}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[english]{babel}

%\usepackage{lipsum} %<--- forgot to actually load resp. make clear the package is needed. Or was commented out in an earlier session.

\title{a nice title}
\author{John Doe}
\date{\today}

\begin{document}

\maketitle

\chapter{Why the sun is hot}
\lipsum %<--- first call of macro, but it's unknown to the compiler so an error is thrown. the usual behaviour.

\chapter{Why it's a bad idea to touch the sun}
\section{The temperature of the sun}
\lipsum

\end{document}


the package lipsum in this case was forgotten to be made known. Usual behaviour in this case: throw an error.

That behaviour causes the author to fix the problem and restart the compile, all of which is timeconsuming.

Now: how would it be possible to have the compiler conviently load the missing packages, on the first parse of the keyword (in our example: \lipsum), and continue compiling without throwing an error and interrupt compiling the document?

• So, it should be able to recognize that some command is defined by some package and then load that package on the fly? This would be a miracle if possible. Lets wait the experts. – Sigur Jun 2 '18 at 22:03
• What about two different packages define same \foo command? – Sigur Jun 2 '18 at 22:09
• If you create a package, which recognizes about 40,000 commands and all packages connected with them, it will be a useful work. – Przemysław Scherwentke Jun 2 '18 at 22:31
• You can't. It is too late. As the question is stated, the answer is that this cannot be done. You cann't load a package after \begin{document} and you cannot know which commands are used in the body of the document until after \begin{document}. Hence, this can't be done during compilation. TeX doesn't read the file and then compile it. It reads the file as it compiles it. You seem to expect that the compiler already knows the contents of the document in the preamble, but it doesn't. You can input code later, so in simple cases, you could make good the loss by doing what the package ... – cfr Jun 3 '18 at 1:32
• ... does. But this won't simply be a matter of inputting the package code and would, as I say, only work in simpler cases. More realistic would be to parse the document prior to compilation, figure out if everything needed is loaded and add stuff if not. Then you'd 'just' have to deal with all the problems others mentioned above. You've asked this before. How many times do you plan to ask it hoping for an answer you like better? – cfr Jun 3 '18 at 1:38

As this question has been open for a month, thought I'd have a go. From the description in the question:

That behaviour causes the author to fix the problem and restart the compile, all of which is timeconsuming.

I take the real question to be about the human cost of having to manually fix and recompile. (And “compilation” to mean merely the user-visible behaviour of invoking some program— e.g. latexmk or arara—rather than necessarily the pdfTeX (or whatever) program itself.) Under this interpretation, it's definitely possible in principle to save the user some time.

(Of course all things are possible if you're willing to change the *TeX program itself, but for better or worse, that is not much done in the TeX world.)

At the bottom of this answer is a proof-of-concept Python script. Suppose you put it into a file called invoke.py, say. Then instead of pdflatex bad.tex you can do python3 invoke.py bad.tex, and you will see the following output in your terminal:

The control sequence \lipsum was undefined
Adding package lipsum to the file.
Great, nothing was undefined this time. This was the output:
This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017) (preloaded format=pdflatex)
restricted \write18 enabled.
entering extended mode
LaTeX2e <2017-04-15>
Babel <3.10> and hyphenation patterns for 84 language(s) loaded.
(/usr/local/texlive/2017/texmf-dist/tex/latex/base/article.cls
Document Class: article 2014/09/29 v1.4h Standard LaTeX document class
Output written on bad.pdf (3 pages, 36350 bytes).


The first three lines come from the Python script. After you've run this, the file bad.tex will have changed to contain the \usepackage{lipsum} line in it. As the whole thing takes 0.47 seconds (and in general TeX is very fast these days), I consider it to genuinely solve the problem (allows the user to avoid the time-consuming step of adding the package to the file manually) even though behind-the-scenes, the TeX program has been invoked multiple times. (Actually even if TeX takes a very long time, the fact is that the program can add the missing packages faster than the human can, and the human would have to re-run the program anyway, so it's always faster.)

This is the Python script invoke.py:

import subprocess
import sys

giant_map_of_macros_to_packages = {
'lipsum': 'lipsum',
# ...
}

"""Inserts a usepackage line into the file."""
with open(filename) as f:
where = contents.find('\\begin{document}')
with open(filename, 'w') as f:
f.write(contents[:where])
f.write('\\usepackage{%s}\n' % package)
f.write(contents[where:])

if __name__ == '__main__':
filename = sys.argv[1]
run_again = True
while run_again:
run_again = False
completed = subprocess.run(['pdflatex', '-halt-on-error', filename],
stdout=subprocess.PIPE)
lines = completed.stdout.decode('utf-8').splitlines()
try:
where = lines.index('! Undefined control sequence.')
line = lines[where + 1]
undefined = line[line.rfind('\\') + 1:]
print('The control sequence \\%s was undefined' % undefined)
package = giant_map_of_macros_to_packages[undefined]
print('Adding package %s to the file' % package)
run_again = True
except ValueError:
print('Great, nothing was undefined this time. This was the output:')
print('\n'.join(lines))
except KeyError:
print('Not sure what package defines %s. Add it to giant_map_of_macros_to_packages' % undefined)


Obviously:

• This is just a proof of concept (to show that it is possible); the right place for this is probably one of the standard LaTeX build tools like latexmk or arara.
• [IMO an even better place would be to add such features to the TeX programs themselves, but that is never going to happen.]
• This assumes things (that are generally true), like that there exists a \begin{document} line in the file before which it is safe to insert a \usepackage{...} line, that the undefined macro is everything after the last \\ on the line following the ! Undefined control sequence. message, etc.
• Someone has to maintain the giant_map_of_macros_to_packages — although in principle that too is automatable from within TeX. (E.g. using LuaTeX: for each package, compare the hash table of macros before and after a package is loaded.) It would also be good to gracefully handle the case when multiple packages define the same macros.
• probably easier to access a database instead of having to maintain a large list with all packages. but I like this PoC. I leave this unaccepted for the time being, in case someone else comes up with a solution. – naphaneal Jul 2 '18 at 10:15
• @naphaneal I'm not sure I understand the distinction you're making, but ok. :-) (Whether the data is stored in code, a flat file, or some more sophisticated database is a detail of how the data is organized (where yes some choices are better than others), and is orthogonal to the fact that the data needs to be generated and maintained. That is, someone needs to maintain a large list with all packages one way or the other; it's independent of whether we access a database.) – ShreevatsaR Jul 2 '18 at 14:58
• just that a SQL query can be outsourced code. but yes, it still needs manpower to be created and maintained, which can be easily split as tasks for multiple individuals, writing several lines of database entries rather than one indidivual writing one large line. anyway, I'm splitting hairs now. – naphaneal Jul 2 '18 at 15:45