5

Problem Description

I am trying to setup latexmk using a .latexmkrc configuration file for "precompiling" the preamble of my documents, and reusing that during reruns.

According to a comment by David Carlisle, using precompliation doesn't really save enough time to be worth the bother [he] wrote mylatex.ltx back in 1988.

However, I have a lot of externalized graphics in my documents, and references inside pgfplots, so the loading times grow quite fast in my case, and I want to explore how much precompilation could speed up the entire process.

Requirements

My current requirements are:

  1. preamble and document must be able to reside inside the same document, for this question let it be mwe.tex.
  2. All compilation must be done from one single call to latexmk.

Literature Review

In particular, I have looked into the following questions and answers:

  • https://tex.stackexchange.com/a/37730 is a bit dated, but the general principle can be taken to more recent versions of LaTeX. The main problem with this proposed solution is, that as stated, [y]ou'll need to run pdflatex -ini ... manually the first time, to create the .fmt file, which contradicts my second requirement.

  • https://tex.stackexchange.com/a/269052 shows how the preamble my be dumped from one single file, which is

    pdflatex -ini \&pdflatex mylatex.ltx mwe.tex
    

    In particular, this is helpful in view of requirement 1.

  • After own research, I have found that a system call

    pdflatex -ini \&pdflatex mylatexformat.ltx mwe.tex
    

    would be the more recent way to do this.

My custom minimal non-working example

Using the previous information and shortcomings in view of my requirements, I intend so sole this problem in the following steps:

  1. Modify the $pdflatex variable in .latexmkrc to require a precompiled preamble mwe.fmt file.
  2. Use add_cus_dep('tex', 'fmt', 1, 'precompilepreamble'); to add a dependency.
  3. Define the perl subroutine precompilepreamble to create the mwe.fmt file if necessary.

I expect the program to behave in the following way:

  1. Call latexmk, which loads my custom .latexmkrc configuration.
  2. First run of pdflatex fails, and will log the missing required file mwe.fmt.
  3. latexmk decides an additional run is needed, and will look up the dependencyies for .fmt.
  4. Accordingly, latexmk will run the custom rule I added previously to generate mwe.fmt.
  5. Afterwards, latexmk will run my modified $pdflatex again and continue sucessfully.

Attached you find the required files:

mwe.tex

\documentclass{scrbook}

\usepackage{pgfplots}
\usepackage{amsmath}
\usepackage{siunitx}
\usepackage{lipsum}


\begin{document}
\title{Shakespeare's Externalize}
\subtitle{To externalize or not to externalize?}
\subject{Case Studies}
\date{\today}
\maketitle{}

\lipsum[1-20]

\end{document}

.latexmkrc

# ensure_path( 'TEXINPUTS', './src/' );
# ensure_path( 'TEXINPUTS', '../src/' );

# Set the program used to generate the PDF
# 1: pdflatex
# 2: postscript conversion, don't use this
# 3: dvi conversion, don't use this
# 4: lualatex
# 5: xelatex
$pdf_mode = 1;

# Add custom dependency.
# latexmk checks whether a file with ending as given in the 2nd
# argument exists ('toextension'). If yes, check if file with
# ending as in first argument ('fromextension') exists. If yes,
# run subroutine as given in fourth argument.
# Third argument is whether file MUST exist. If 0, no action taken.
add_cus_dep('tex', 'fmt', 1, "compilepreamble");

# show custom dependencies for debugging
show_cus_dep();

# parse missing files
push @file_not_found, '^I can\'t find the format file `(.*)\'\!';

sub compilepreamble {
    my ( $base) = @_;
    # just output a simple debug string to console
    print ("**** compilepreamble: ( $base ) ****\n");

    # create the fmt file
    $ret = system("pdflatex",
        "-jobname", "$base",
        "-ini",
        "&pdflatex" ,
        "mylatexformat.ltx",
        $base
         );

    # if there were errors, return the error codes
    return $ret;
}


# modify $pdflatex to use custom format
$pdflatex = 'pdflatex  -fmt mwe %B %O %S';



Observed Behaviour

Instead of pdflatex failing, it tries to run mktexfmt:

$  latexmk -g
Custom dependency list:
  tex fmt 1 compilepreamble
Rc files read:
  .latexmkrc
Latexmk: This is Latexmk, John Collins, 7 Jan. 2023. Version 4.79.
Force everything to be remade.
Latexmk: applying rule 'pdflatex'...
Rule 'pdflatex':  Reasons for rerun
Category 'other':
  Rerun of 'pdflatex' forced or previously required

------------
Run number 1 of rule 'pdflatex'
------------
------------
Running 'pdflatex -fmt mwe   -recorder  "mwe.tex"'
------------
This is pdfTeX, Version 3.141592653-2.6-1.40.24 (TeX Live 2022) (preloaded format=mwe)
 restricted \write18 enabled.

kpathsea: Running mktexfmt mwe.fmt
mktexfmt: mktexfmt is using the following fmtutil.cnf files (in precedence order):
...
I can't find the format file `mwe.fmt'!
Latexmk: fls file doesn't appear to have been made.
Latexmk: Getting log file 'mwe.log'
Latexmk: Couldn't read log file 'mwe.log':
  No such file or directory

Question

  1. How can I prevent latexmk, or rather, pdflatex from trying to run mktexfmt?
  2. The line I can't find the format file `mwe.fmt'! is only output to console and no mwe.log is generated, which makes reruns impossible, and does not detect my custom dependency. How can I change this behaviour?
  3. I could inject a custom subroutine into $pdflatex, which would then check if the format file exists, and possibly create it if not. This could even be combined with a hashing process, to ensure only recompilation if needed and ... oh my, I notice I am copying the functionality of latexmk. So, is there a way to do this in latexmk in the first place?

I'm also looking forward to hearing other suggestions.

4
  • pdftex has [-no]-mktex=FMT disable/enable mktexFMT generation (FMT=tex/tfm/pk) commandline option (see pdftex --help) Jul 5, 2023 at 16:28
  • That's interesting. I seemingly fail to understand this command. When I run pdflatex --no-mktex=tex --fmt=mwe mwe.tex (which I believe is what @DavidCarlisle suggested), it seems that options -fmt and --no-mktex are exclusive/overriding each other. I couldn't find documentation on this. Just to clarify, I want latexmk to try to use the format on the first run and just fail with appropriate error in log file.
    – marc
    Jul 5, 2023 at 17:04
  • pdftex --no-mktex=fmt --fmt mwe seems to work, i get I can't find the format file `mwe.fmt'! Jul 5, 2023 at 17:29
  • Thank you for the helpful comments, I was not noticing it should be --no-mktex=fmt, this choice was not documented.
    – marc
    Jul 6, 2023 at 13:23

2 Answers 2

3
  1. How can I prevent latexmk, or rather, pdflatex from trying to run mktexfmt?
$ pdftex --no-mktex=fmt --fmt mwe file
This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) (preloaded format=mwe)
 restricted \write18 enabled.
I can't find the format file `mwe.fmt'!

shows a run that does not trigger mktexfmt

  1. The line I can't find the format file `mwe.fmt'! is only output to console and no mwe.log is generated, which makes reruns impossible, and does not detect my custom dependency. How can I change this behaviour?

I don't think that is changeable

  1. I could inject a custom subroutine into $pdflatex, which would then check if the format file exists, and possibly create it if not.

You could, but (without the option in (1.) above) pdftex is already doing that check and calling mktexfmt in the case the file does not exist so a simpler approach is probably to configure mktexfmt so that it builds mwe.fmt when requested. (texdoc mktexfmt has some information on how to set up the fmtutil config)

1
  • Thank you for the answer, however, reading it and trying out your suggestions I found an answer that seems to fit even better, which I posted as well.
    – marc
    Jul 6, 2023 at 13:24
2

The excellent answer by David Carlisle has lead me to discovering a way how my goals can be achieved using pure latexmk, so I am going to share my solution and see how it might be able to be improved.

Solution

As noted by David Carlisle,

  $ pdftex --no-mktex=fmt --fmt mwe file 
  This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) (preloaded format=mwe)
    restricted \write18 enabled. 
  I can't find the format file `mwe.fmt'!

shows a run that does not trigger mktexfmt

and it seems I was not able to read the document right to put in the option --no-mktex=fmt.

As noted by David Carlisle as well, this run does not log to a respective log file, so no rerun for latexmk is triggered.

My solution revolves around chenging the output stream from stdout to a logfile, which can be entirely achieved in .latexmkrc:

mwe.tex

\documentclass{scrbook}

\usepackage{pgfplots}
\usepackage{amsmath}
\usepackage{siunitx}
\usepackage{lipsum}
\usepackage{geometry}


\begin{document}
\title{Shakespeare's Externalize}
\subtitle{To externalize or not to externalize?}
\subject{Case Studies}
\date{\today}
\maketitle{}

\lipsum[1-30]

\end{document}

.latexmkrc

# Set the program used to generate the PDF
# 1: pdflatex
# 2: postscript conversion, don't use this
# 3: dvi conversion, don't use this
# 4: lualatex
# 5: xelatex
$pdf_mode = 1;

# Add custom dependency.
# latexmk checks whether a file with ending as given in the 2nd
# argument exists ('toextension'). If yes, check if file with
# ending as in first argument ('fromextension') exists. If yes,
# run subroutine as given in fourth argument.
# Third argument is whether file MUST exist. If 0, no action taken.
add_cus_dep('tex', 'fmt', 1, "compilepreamble");

# show custom dependencies for debugging
show_cus_dep();


# parse missing files
push @file_not_found, '^I can\'t find the format file `(.*)\'\!';


# modify all tex engines to run with my customized instruction set
# in particular:
#   - suppress mktexfmt using --no-mktexfmt=fmt
#   - request use of custom format using --fmt=%B
#   - call additional options provided to latexmk using %O
#   - call the source file %S
set_tex_cmds("--shell-escape  --interaction=batchmode  --file-line-error --synctex=1 --no-mktex=fmt --fmt=%B  %O %S > %B.log");



# custom sub handling the loading of the preamble and dumping it into $base.fmt
sub compilepreamble {
    my ( $base) = @_;
    # just output a simple debug string to console
    print ("**** compilepreamble: ( $base ) ****\n");

    # create the fmt file
    $ret = system("pdflatex",
        "-jobname", "$base",
        "-ini",
        "&pdflatex" ,
        "mylatexformat.ltx",
        $base
         );

    # if there were errors, return the error codes
    return $ret;
}

Comments

  1. My solution seems create the fmt on first run, and reuse it in subsequent runs.
  2. My solution does trigger a rebuild of the mwe.fmt file each time one calls latexmk after the file mwe.tex has changed. It does not make a distinction, whether the change has been made in the document or the preamble. This is a bit of a shortcoming, however, in my usecase it can be neglected, as
  • most content resides in their own files and is placed using \input{}
  • the relevant savings in compilation time will occurr during reruns for externalization of graphics.

If anyone wants to suggest improvements, I'll gladly hear them.

4
  • +1 for working code rather than random untested thoughts in the other answer:-) Jul 6, 2023 at 13:40
  • Would it be possible to have this solution work with custom output directory (out_dir) or auxiliary directory (aux_dir)? Somehow it only works when the files are in the same directory Nov 24, 2023 at 13:12
  • @JoanMarcèiIgual I have been experimenting with custom output directories and auxiliary directories as well, but unfortunately have not come up with a satisfactory solution. The main obstacle is, that format files *.fmt added through dependencies must currently reside next to their source files (see my question here: tex.stackexchange.com/a/690540/53868). In my opinion, this completely circumvents separation of code and artefacts, so I no longer pursue the idea of using latexformat ("precompiling" also does not work with lualatex).
    – marc
    Nov 30, 2023 at 10:38
  • @JoanMarcèiIgual If you still want to investigate possible workarounds, you need to tell kpathsea where to look for the format files using ensure_path('TEXFORMATS', "./$source_dir//"); or similar. If you create a proper question from it, I can post my working (hackish) solution as an answer.
    – marc
    Nov 30, 2023 at 11:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .