compiling pdftex 2.2 seconds with 2012 vs 29 seconds with 2018 version

Question

I'm not quite sure where to start looking to fix this. I have recently migrated from an old Ubuntu 14.04 box using texlive/2012 to an 18.04 box running texlive-2018.

old box

pdfTeX 3.1415926-2.4-1.40.13 (TeX Live 2012)
kpathsea version 6.1.0
Copyright 2012 Peter Breitenlohner (eTeX)/Han The Thanh (pdfTeX).
There is NO warranty.  Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Peter Breitenlohner (eTeX)/Han The Thanh (pdfTeX).
Compiled with libpng 1.5.10; using libpng 1.5.10
Compiled with zlib 1.2.7; using zlib 1.2.7
Compiled with xpdf version 3.03

new box

pdfTeX 3.14159265-2.6-1.40.19 (TeX Live 2018)
kpathsea version 6.3.0
Copyright 2018 Han The Thanh (pdfTeX) et al.
There is NO warranty.  Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Han The Thanh (pdfTeX) et al.
Compiled with libpng 1.6.34; using libpng 1.6.34
Compiled with zlib 1.2.11; using zlib 1.2.11
Compiled with xpdf version 4.00

The tex file contains quite a number of images which is where the slowdown seems to occur (compiling files without images is quite fast)

I'm passing in a format file, but, basically both jobs are called like this:

pdfetex --output-format=pdf -fmt=$fmt_path -no-shell-escape -output-directory=$out_dir filename.tex

Does anyone have hints on where to start looking to track this down?

I also zipped up the 2012 directory from the old box, and moved it onto the new box, and ran that. the job completed in 2.2 seconds on the new box. So it's not the disk or machine, it's something to do with 2012 vs 2018.

What I can see, in the logs, is that the 2018 version is just a lot slower processing the images. At least, with verbose turned on I can see it's taking ~600ms per image or so.

Sorry for a pretty wooly question, but, any help would be most appreciated!

Update - 20th August 2018

I have done some more investigation.

1. It's not png's which are the issue. I created a document containing just pngs (we have a few different figure types which are included) and they both rendered in a similar speed on both 2012 and 2018 versions.
2. I created a document containing mostly broken image links. This one was interesting as it took about 0.2 seconds to build the broken document using 2012, but almost 8 seconds to build the same broken document on 2018. This suggests there's something very inefficient going on regarding the searching of files.

Anyway, continuing to investigate.

Kpath debugging

I don't think this is actually it, but, here's a difference in logging out the searching between 2012 and 2018 (for the same file)

2012

(apppath/texinput/tex/EA910706.FIG)kdebug:hash_lookup(openin_any../acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(openin_any) => a
kdebug:hash_lookup(try_std_extension_first../acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(try_std_extension_first) => t
kdebug:hash_lookup(openin_any../acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(openin_any) => a
kdebug:hash_lookup(try_std_extension_first../acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(try_std_extension_first) => t

2018

(apppath/texinput/tex/EA910706.FIG)kdebug:hash_lookup(openin_any.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(openin_any) => a
kdebug:hash_lookup(try_std_extension_first.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(try_std_extension_first) => t
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1
kdebug:hash_lookup(openin_any.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(openin_any) => a
kdebug:hash_lookup(try_std_extension_first.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(try_std_extension_first) => t
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1
kdebug:hash_lookup(texmf_casefold_search.formatpath/acc4-pdf.fmt) => (nil)
kdebug:hash_lookup(texmf_casefold_search) => 1

Answer 1

The culprit for us turned out to be the case (insensitive) searching that has become the default for kpathsea in 2018. Thanks to @moewe for the pointer.

The fix was to add to our texmf.cnf file:

texmf_casefold_search = 0

After that, all of our pdftex jobs compiled at the speeds we were used to.