Does LaTeX have to reprocess included files that haven't changed?

I have included several .tex files (they do not contain the header code like documentclass, begin and end document, etc). When I include the file using \include everything works. But if I recompile the code I still takes a while even though most of the includes(or even all) have not changed.

My basic method to deal with this is to comment out the includes I do not want compiled but I see no reason why they can't be "precompiled" and reused each time unless the .tex file changed.

-
I am afraid, it can't be the case. Say you are editing one file and you add some references there. Unless you reset these counters per chapter (something like that) the numbers will change. Hence LaTeX has to recompile everything. If it is draft, then you can consider includeonly{filename1,filename2...}. – Harish Kumar May 13 '12 at 5:06
@HarishKumar Yeah, but what if I don't care about the references updating unless I specifically ask for it? It's easier for me to use % on the include's than includeonly. The reference updates are really not that important since I don't use many. – Uiy May 13 '12 at 6:04
The fact that the .tex file isn't changed doesn't mean that its typeset version will stay the same. Cross-references in it may not resolve correctly when other files are changed; a call of \today might produce a different date; a change in the definition of a command in the root file might modify drastically the output. Would your process take care of all this? – egreg May 14 '12 at 6:42

TeX works in a linear fashion with an input file, and while there are methods to split up in input (ultimately based on the \input primitive), these are for the author's convenience only. TeX still processes all of the input in a linear manner: once a file is \input it is as though the material was all in one big file.

At the same time, it is possible for changes in one part of a document to affect another (for example, altering the set up of the running headers or page numbering). So it is not possible to simply say 'I can skip this input'. Writing the output is also a linear process: there is no direct link between individual .tex files and parts of the .dvi or .pdf in a way that would allow TeX to take 'old' pages and 'recycle' them.

The LaTeX \include mechanism does have features to help with larger documents. Using \includeonly it is possible to compile only part of a document, while retaining cross-references and so on. This however works by entirely skipping the files which are not included, so you do not get the complete version of your document.

There are also methods to precompile the preamble of a document into a custom format, and then only retypeset the document body. See for example mylatexformat.

-

There is no existing way to do this. The only way is using \include and \includeonly (or \excludeonly) and simply skip certain chapters.

However, it would be possible to produce PDFs from \include files and include these using \includepdf if the files didn't changed. I already wrote code like this for other applications: standalone can compile it's own document in the background to get access to the resulting PDF, see Shell escape after PDF is finished and closed? and especially Can one TeX file output to multiple PDF files? (both by Ulrike Fischer). File changes could be checked using my filemod package: Can (La)TeX compare the time stamps of two external files?.

So a new package could redefine \include to produce PDFs only if the .tex file changed. Handling changes in sub-files and images are more difficult. Then the document is assembled using pdfpages. However, a major drawback is that all PDF annotations like hyperlinks will be gone, because pdftex doesn't support them in included PDF files. Therefore this should only be used as a form of draft mode.

An alternative but similar approach would be to have a script or Makefile which checks for file changes and compiles the document for every chapter by using:

pdflatex -jobname "chapt1" "\includeonly{chapter1}\input{main}"

Then you can view the PDFs directly. It is also possible to have a main PDF which bookmarks point to other PDFs (holding the chapters), so you can browse through the whole document. This also allows hyperlinks etc. in the chapter PDFs.

-
Could I someway use lualatex to extract the pages that from the \include'ed file and the insert those pages back if the file hasn't changed? My included files cannot stand alone(no document class or usepackages and other setup stuff. Even if the outputs were just jpgs it would work. In fact, why couldn't you "recompile" if a new reference was made to the included file? It seems like there is no real reason why it shouldn't work? – Uiy May 13 '12 at 15:48
@Uiy: If you make a copy of the PDF of the last run and the get the starting and end pages of the \include files from the .aux files then you could \includepdf them. This is better done by an external script or Makefile. – Martin Scharrer May 13 '12 at 17:47
I think lua can handle it just find and it doesn't need to be an external script. The problem I see is that copying the pdf isn't going to work for multiple includes when some will change and others won't unless you make a bunch of copies(one for each include). Is there any way to delete the pages in the copy of the pdf to match only those in the aux? so it will represent no more than what it needs? – Uiy May 13 '12 at 18:16
@Uiy: You can cut and reorder a PDF with tools like pdftk in any way you want. As I see it, you only need to have one PDF copy and the physical page numbers from the .aux file. Then redefine \include to \includepdf the pages from the old PDF if the file didn't changed or \include the source if it did. – Martin Scharrer May 13 '12 at 21:04

I don't think that this approach can lead to result that are applicable to a wide range of documents.

You're starting from a false assumption, in my opinion: the fact that a .tex file isn't changed doesn't mean that its typeset version will stay the same, for a number of reasons.

1. Cross-references in it may not resolve correctly when other files are changed.

2. A call of \today might produce a different date.

3. A change in a previous .tex file may produce different chapter or section numbering.

4. A change in a previous .tex file may produce different page numbers.

5. Most importantly, a change in the definition of a command in the root file might modify drastically the output even if no numbering has changed.

Would your process take care of all this? Documents where such constraints are not relevant are better managed as separate complete TeX documents to be merged with an external utility such as pdftk. Of course one can write a Lua script that wraps compilation of such documents and does the merge.

-
No, I just don't care. The biggest thing I want to do is avoid the delay(several minutes) for each include and don't want to have to use some way to comment them out. It will not be hard to do a clean and force a complete refresh. Your assumption is that all the includes much change. In fact though one should then be able to extend my idea to make it even more robust. Say when a reference is created or changed, or some include before another include is changed then one can simply regenerate the pdfs. Your assumption is that it can't be done. I believe it can... just a matter of how much work. – Uiy May 14 '12 at 8:42
maybe in the future I'll realize it's not as good as I think but for my current document I spend several minutes waiting on several \include's to compile because they generate a lot of graphics programmatically(tiks + lua). With my code(although not working at this point) I can precompile the includes and convert \include behind the scene to \includepdf when the tex file hasn't changed. This will drastically speed of the compilation. In the future I might be able to determine if one include effects another and recompile that affect include too. – Uiy May 14 '12 at 8:45
Basically right now I simply determine if a tex file has changed by checking it's stored modified time compared with it's current modified time. One can potentially extend this logic to cover the issues you have talked about(references, order, etc...). I could also use sub includes around just the graphics generation code and have those compiled to pdf's(although that won't work because they are not each there own page it might be designed to work with tikzexternalize). – Uiy May 14 '12 at 8:47
@Uiy No, it wouldn't work. An \included file generates an .aux file where the labels defined in it are stored; but there's no way to know what references are used in the file that may depend on changes in other files, unless one radically alters the way \ref works. – egreg May 14 '12 at 8:53
Huh? All one would have to do would be to track ref changes. Modify ref so that when it is called it checks the value passed with a stored version. If it is the same then the ref hasn't changed and no reason to update. This is what I'm basically doing with include. I'm probably going to do the same with tikz pictures. (although in my case it is easy because of how I generate the pictures) The whole idea is simple to compare the old with the new. The problem is figuring out what is old and what is new. Somethings are easier than others. With \ref it might require a lot of work but possible – Uiy May 14 '12 at 9:00