Pdflatex on file with large amount of pictures

Question

I have a tex file that's generated by an automated bioinformatics pipeline as a report. Usually, this is roughly 50-150 pages, but that's based on 1-3 samples. It contains many quality control graphs. I now have a run with 60 samples, which explodes the size to 4026 pages. The problem is that pdflatex keeps failing when I include pictures. There are 8843 pictures in the file, in 4874 figure blocks and they are all called within a minipage and subfloat, e.g. something like:

    \begin{figure}[h!]
    \centering
    \begin{minipage}[b]{0.48\textwidth}
        \centering
        \subfloat[Raw read 1]{
            \includegraphics[width=\textwidth]{/data/DIV4/ihb-diabetes/projects/slik-tcell/analysis/01_gentrap_new/sample_H7-59-17-AN/lib_S61-L003/flexiprep/H7-59-17-AN_S61_L003_R1_001.fastqc/H7-59-17-AN_S61_L003_R1_001_fastqc/Images/sequence_length_distribution.png}
        }
    \end{minipage}
    \begin{minipage}[b]{0.48\textwidth}
        \centering
        \subfloat[Preprocessed read 1]{
            \includegraphics[width=\textwidth]{/data/DIV4/ihb-diabetes/projects/slik-tcell/analysis/01_gentrap_new/sample_H7-59-17-AN/lib_S61-L003/flexiprep/H7-59-17-AN_S61_L003_R1_001.qc.fastqc/H7-59-17-AN_S61_L003_R1_001.qc_fastqc/Images/sequence_length_distribution.png}
        }
    \end{minipage}
    \caption{Read length distribution for read 1.}
    %\label{fig:length_dist_before_and_after_1}
\end{figure}

I keep getting these two errors:

!pdfTeX error: /usr/bin/pdflatex (file /data/DIV4/ihb-diabetes/projects/slik-tc
ell/analysis/01_gentrap_new/sample_B7-59-0-AM/lib_S46-L002/flexiprep/B7-59-0-AM
_S46_L002_R2_001.fastqc/B7-59-0-AM_S46_L002_R2_001_fastqc/Images/per_sequence_q
uality.png): buffer overflow at file ../../../texk/web2c/pdftexdir/utils.c, lin
e 183
 ==> Fatal error occurred, no output PDF file produced!

and:

</data/DIV4/ihb-diabetes/projects/slik-tcell/analysis/01_gentrap_new/sample_B7-
59-0-AM/lib_S46-L002/flexiprep/B7-59-0-AM_S46_L002_R2_001.qc.fastqc/B7-59-0-AM_
S46_L002_R2_001.qc_fastqc/Images/per_sequence_gc_content.png, id=55775, 803.0pt
 x 602.25pt>
! TeX capacity exceeded, sorry [pool size=6181497].
\Ginclude@png ...pty \else , page \GPT@page \fi >}
                                              \GPT@info {#1\ifx \GPT@pag...
l.144122             }

!  ==> Fatal error occurred, no output PDF file produced!

This happens even when I try to increase pool_size and extra_mem_bot. My final pdflatex call was:

'/usr/bin/pdflatex' -8bit --extra-mem-bot=11150000000 --pool-size=196181497  '-output-directory' '/data/DIV4/ihb-diabetes/projects/slik-tcell/analysis/01_gentrap_new/report'  '-jobname' 'gentrap_report'  '/data/DIV4/ihb-diabetes/projects/slik-tcell/analysis/01_gentrap_new/gentrap_report.tex'

The only way I managed to get a PDF file was by running pdflatex with another user which doesn't have permissions to read the folders where the pictures are located. Naturally, that resulted in a whole load of permission denied errors, but it got me something. That leads me to believe the errors are figure related. Would there be a way which reduces the memory load of my figures? (Whether a 4026 page report with nearly 9000 figures is a readable document is another question altogether).

Answer 1

It is the the length of the paths you are using.

\documentclass{article}
\batchmode
\usepackage{graphicx}
\newcount\zz
\begin{document}
\def\z{}
\loop
\edef\z{\z/}
\includegraphics{/c/tmp/../tmp/../tmp/../tmp/../tmp/../tmp/../tmp/../tmp/../tmp/../tmp/..%
/tmp/../tmp/../tmp/../tmp/../tmp/../tmp/../tmp/../tmp/\z house.png}\par
%\includegraphics{/c/tmp/\z house.png}\par
\advance\zz1
\ifnum\zz<1500
\repeat

\end{document}

Produces the error that you show, in my case

! TeX capacity exceeded, sorry [pool size=6135010].

the ../tmp/...... construct just accesses the file /c/tmp/house.png but with a suitably long path and the \z redefinition ensures that a distinct path is used every time.

Every distinct path uses up the string pool.

If you comment out the \includegraphics with the long path and use instead the one with a shorter path then the document succeeds with pool usage shown as

 5790653 string characters out of 6135010

In your case you could use

\includegraphics{sequence_length_distribution.png}

which would drastically reduce the string pool usage, you just need to ensure that the directory with the images is in the TEXINPUT path.

For example if they were in the current directory, or just set

TEXINPUTS=/data/DIV4/ihb-diabetes/projects/slik-tcell/analysis/01_gentrap_new/sample_B7-59-0-AM/lib_S46-L002/flexiprep/B7-59-0-AM_S46_L002_R2_001.fastqc/B7-59-0-AM_S46_L002_R2_001_fastqc/Images/:

before running latex.