12

I've done some major changes in my document (dissertation, nearly 300 pages) and now compiling time has increased nearly by factor 3!

One pdflatex run took normally less than 3 minutes before and now takes about 6 minutes, for complete compiling with latexmk it took 26 minutes!!!

What I've changed:

  • switched all source files (.tex and .bib) to UTF8
  • had to switch from bibtex/biblatex to biber/biblatex

My questions:

  • Can the change to UTF8 slow down compiling?
  • I read about some problems with kpathsea checking deeps directory paths during each compiling. How can I find out if I have such a problem?
    I formerly had 2 TeXLive distros on my machine, but currently only have TeXLive 2014 installed.

Hard and Software

  • MacBook 13" (2007)
  • MacOS X 10.6.8
  • TeXLive 2014
  • Aquamacs 2.5 as editor
  • compiling is currently done from command line (pdflatex or latexmk)

Further investigations of the suspects:

1. TEXINPUTS

the command kpsexpand \$TEXINPUTS gives the following result: (added linebreaks after each path)

.:{/Users/mbg4/Library/texlive/2014/texmf-config,
 /Users/mbg4/Library/texlive/2014/texmf-var,
/Users/mbg4/Library/texmf,
!!/usr/local/texlive/2014/texmf-config,
!!/usr/local/texlive/2014/texmf-var,
!!/usr/local/texlive/texmf-local, 
!!/usr/local/texlive/2014/texmf-dist}/tex/{kpsewhich,generic,}//

The directory /Users/mbg4/Library/texmf does not exist. However, there is a directory /Users/mbg4/Libary/texlive with the following content

texlive mbg4$ ls -ld $(find .)
drwxrwxr-x  6 mbg4  mbg4      204 16 Jan 12:16 .
-rw-r--r--@ 1 mbg4  mbg4     6148  1 Mär 12:50 ./.DS_Store
drwxrwxr-x  4 mbg4  mbg4      136 18 Mär  2011 ./2010
-rw-r--r--@ 1 mbg4  mbg4     6148  1 Mär 12:50 ./2010/.DS_Store
drwxrwxr-x  6 mbg4  mbg4      204 29 Jul  2011 ./2010/texmf-var
-rw-r--r--@ 1 mbg4  mbg4     6148  1 Mär 12:50 ./2010/texmf-var/.DS_Store
drwxrwxr-t  3 mbg4  mbg4      102 29 Jul  2011 ./2010/texmf-var/fonts
drwxrwxr-t  3 mbg4  mbg4      102 29 Jul  2011 ./2010/texmf-var/fonts/tfm
drwxrwxr-t  3 mbg4  mbg4      102 29 Jul  2011 ./2010/texmf-var/fonts/tfm/public
drwxrwxr-t  2 mbg4  mbg4       68 29 Jul  2011 ./2010/texmf-var/fonts/tfm/public/pandora
drwxrwxr-x  4 mbg4  mbg4      136 18 Mär  2011 ./2010/texmf-var/luatex-cache
-rw-r--r--@ 1 mbg4  mbg4     6148  1 Mär 12:50 ./2010/texmf-var/luatex-cache/.DS_Store
drwxrwxr-x  4 mbg4  mbg4      136 18 Mär  2011 ./2010/texmf-var/luatex-cache/context
-rw-r--r--@ 1 mbg4  mbg4     6148  1 Mär 12:50 ./2010/texmf-var/luatex-cache/context/.DS_Store
drwxrwxr-x  4 mbg4  mbg4      136 18 Mär  2011 ./2010/texmf-var/luatex-cache/context/e9159e878e989105b2ef2d27fad97515
-rw-r--r--@ 1 mbg4  mbg4     6148  1 Mär 12:50 ./2010/texmf-var/luatex-cache/context/e9159e878e989105b2ef2d27fad97515/.DS_Store
drwxrwxr-x  2 mbg4  mbg4       68 11 Feb  2011 ./2010/texmf-var/luatex-cache/context/e9159e878e989105b2ef2d27fad97515/trees
drwxrwxr-x  3 mbg4  mbg4      102 18 Dez  2010 ./2010/texmf-var/texdoc
-rw-r--r--  1 mbg4  mbg4   818250 10 Jul  2011 ./2010/texmf-var/texdoc/cache-tlpdb.lua
drwxrwxr-x  4 mbg4  mbg4      136  1 Mär 12:50 ./2011
-rw-r--r--@ 1 mbg4  mbg4     6148  1 Mär 12:50 ./2011/.DS_Store
drwxrwxr-x  3 mbg4  mbg4      102 15 Dez  2011 ./2011/texmf-var
drwxrwxr-x  3 mbg4  mbg4      102 15 Dez  2011 ./2011/texmf-var/texdoc
-rw-r--r--  1 mbg4  mbg4   879199 15 Dez  2011 ./2011/texmf-var/texdoc/cache-tlpdb.lua
drwxrwxr-x  3 mbg4  mbg4      102 16 Jan 12:16 ./2014
drwxrwxr-x  4 mbg4  mbg4      136 16 Jan 21:32 ./2014/texmf-var
drwxrwxr-t  3 mbg4  mbg4      102 16 Jan 21:32 ./2014/texmf-var/fonts
drwxrwxr-t  3 mbg4  mbg4      102 16 Jan 21:32 ./2014/texmf-var/fonts/pk
drwxrwxr-t  3 mbg4  mbg4      102 16 Jan 21:32 ./2014/texmf-var/fonts/pk/ljfour
drwxrwxr-t  3 mbg4  mbg4      102 16 Jan 21:32 ./2014/texmf-var/fonts/pk/ljfour/public
drwxrwxr-t  3 mbg4  mbg4      102 16 Jan 21:32 ./2014/texmf-var/fonts/pk/ljfour/public/wasy
-rw-r--r--  1 mbg4  mbg4    15908 16 Jan 21:32 ./2014/texmf-var/fonts/pk/ljfour/public/wasy/wasy10.657pk
drwxrwxr-x  3 mbg4  mbg4      102 16 Jan 12:16 ./2014/texmf-var/texdoc
-rw-r--r--  1 mbg4  mbg4  1142858 18 Feb 00:22 ./2014/texmf-var/texdoc/cache-tlpdb.lua

2. Tikz/PGFplots

It's commented out, so I think it should have no impact at all, but I still try to remove it.

  • But: I use IPE 7 for creating my graphics and had tried to embed a tikz tree in an IPE document. I removed the tikz tree, but I forgot to remove the usepackage commands in the IPE latex preamble of this document. And the resulting PDF (created with IPE) is used via includegraphics in my pdflatex document. So I'll see if it helps removing the tikz code from there.

3. Datatool

  • I don't use datatool, at least I'm not aware of it and a grep -ir datatool ...

4. Unicode

not yet investigated

5. biber

  • biber seems to have a large impact. I use this biblatex setup currently:

    \usepackage[natbib=true, style=numeric-comp, backend=biber, defernumbers,useprefix,firstinits=true, maxnames=99, maxcitenames=3]{biblatex}

and as soon as I replace backend=biber to "backend=bibtex8`, the compiling time was reduced dramatically (from nearly 5 minutes for 98 pages to 37 seconds - only 1 pdflatex run).

Switching back to biber, I get a compiling time of 2min40sec - I'm quite confused, maybe some cache issues?

to be continued here: Why does biber increase compilation time of pdflatex runs dramatically (factor 2.8!!)?

Remarks:

  • I know that this is very little information at the moment and I'll do some more tests to find the problem, but some general suggestions and comments how to approach that problem would help me very much!
  • I'm using version control and I'll try to go back to the original state before the compiling time went up and try to compare the outputs.
Improve this question
11
  • 1
    I used the exact same software & bibliography setup and I experienced the same problem when I had some undefined references in my document. Note: if you only address the 2014 version of TL in your editor's paths... it shouldn't be a problem, theoretically speaking. But there certainly will be a more profound response by someone else.
    – henry
    Mar 1, 2015 at 10:28
  • 1
    Mac file manager? Beware Mac OSX has hidden files (I don't have a Mac myself, but a friend of mine had some problems with a TeX installation, and it was hard to find out where all files were located). However try to see what happens when you compile from the command line. Thus you'll detect if the problem comes from the editor configuration or really from the installation.
    – Bernard
    Mar 1, 2015 at 10:29
  • 1
    @Martin Nothing strange in your TEXINPUTS, recursion in ~/Library/texlive/2014/texmf-var is not a problem as the structure is not really so deep.
    – egreg
    Mar 1, 2015 at 12:15
  • 1
    I have had experience with three things slowing down compilation: 1) TikZ, 2) PGFplots with gnuplot, 3) datatool. The latter may be sensitive to your encoding change; I know we use UTF8.
    – Raphael
    Mar 1, 2015 at 13:44
  • 2
    In addition to rolling back on version control, I suspect you'll find that there are a very small number of chapters or sections that are dominating the compile time (and likely dominate the change in compile time). Comment out half your chapters in the main document: does compile time drop by a large percentage or a small one? If small, continue commenting out until you find the chapters that have the largest impact. Then as you start working through commenting sections of that chapter, you'll likely find the bottleneck.
    – Mike Renfro
    Mar 1, 2015 at 15:25

2 Answers 2

Reset to default
9

It seems that using backend=biber is causing the amazing slow down,

I've asked a new focused question here to discuss this issue: Why does biber increase compilation time of pdflatex runs dramatically (factor 2.8!!)?

Performance comparison biber / bibtex8:

Test setup

  • my LaTeX document, limited to one long chapter by \includeonly,
  • 98 pages with many many references, many of them in large tables.
  • large .bib file (5.6 MB) with >3000 references, 350 of them cited in the document
  • I'm compiling from OS X terminal to measure the duration of the compilation:
    latexmk -C; time pdflatex main.tex

Test results

with biber:

  • biber cache removed once with rm -rfbiber --cache``

  • latexmk -C; time pdflatex Diss.tex
    Output written on Diss.pdf (83 pages, 2130562 bytes).
    real 2m17.499s
    user 1m47.626s
    sys 0m1.030s

  • biber Diss
    real 0m27.257s
    user 0m20.568s
    sys 0m1.247s

  • pdflatex Diss.tex Output written on Diss.pdf (95 pages, 2453607 bytes).
    real 4m7.708s
    user 3m27.864s
    sys 0m1.029s

  • again pdflatex Diss.tex
    Output written on Diss.pdf (95 pages, 2452370 bytes). real 4m33.769s user 3m36.491s sys 0m1.252s

with bibtex8

  • remove all aux and bbl files etc.
  • latexmk -C; time pdflatex Diss.tex
    Output written on Diss.pdf (82 pages, 2130383 bytes).
    real 0m48.470s
    user 0m19.161s
    sys 0m0.522s
  • time bibtex Diss
    (There were 200 error messages) sure :-(
    real 0m1.258s
    user 0m0.850s
    sys 0m0.061s

  • time pdflatex Diss.tex
    Output written on Diss.pdf (95 pages, 2452225 bytes).
    real 2m26.433s
    user 1m44.921s
    sys 0m0.932s

  • second pdflatex Diss.tex
    Output written on Diss.pdf (95 pages, 2450936 bytes).
    real 2m27.148s
    user 1m51.958s
    sys 0m0.953s

Improve this answer
7

In absolute terms utf8 does slow down the processing as (if you have any non ascii characters) you have to handle 2 (or more) bytes rather than 1, and it can not possibly take less time to process two bytes than 1 byte. But unless you have millions of non-ascii characters and your document does nothing but write them out to files and read them back in thousands of time, I doubt that it really makes any observable difference.

Disk searching is almost always the culprit.

On my system kpsexpand shows (white space added)

$ kpsexpand \$TEXINPUTS
.:{/home/davidc/.texlive2014/texmf-config,
   /home/davidc/.texlive2014/texmf-var,
   /home/davidc/texmf,
  !!/usr/local/texlive/2014/texmf-config,
  !!/usr/local/texlive/2014/texmf-var,
  !!/usr/local/texlive/texmf-local,
  !!/usr/local/texlive/2014/texmf-dist}
                      /tex/{kpsewhich,generic,}//

Note the !! that means that those directories are pre-indexed into an ls-lR file and tex will not search the filesystem. this means that when you go

 \documentclass{article}

TeX knows where article.cls is and includes it.

If the !! were not there and the ls-r file (texhash command) was missing or out of date then tex would do a recursive walk of the file system in all the above listed places looking for article.cls this can easily take several orders of magnitude longer.

So check you TEXINPUTS and run texhash (or mktexlsr which is more or less same thing)

Improve this answer
3
  • thanks, I've added the correponding output - looks quite similar to yours, is that ok or is there any problem?
    – MostlyHarmless
    Mar 1, 2015 at 11:22
  • @Martin do you have a large texmf tree under /Users/mbg4/Library/texmf (or in any of the directories at the start of the path) if you had an old texlive distrib in there that would be bad:-) in particular look at the log of your document, do most included packages come directories in the from !! trees.
    – David Carlisle
    Mar 1, 2015 at 11:27
  • the above mentioned texmf directory in my home path does not exist, however there are some other Tex paths, more info see my edit in the question - there are also references to older texlive distributions
    – MostlyHarmless
    Mar 1, 2015 at 12:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .