Issue with -blx file path using custom build directory

Question

I know there are plenty of other topics related to this question, however none of the ones I read resolves the issue presented here. I made a minimum working example available here, if you want to reproduce the problem. This post is a revival of an older question of mine[1].

Desired Behaviour

I would like to have a custom build folder that will contain all the PDFLaTeX / LUALaTeX intermediate build (and eventually output) files. Having a project structure resembling something like:

root
┣━━ main.tex
┣━━ figures/
┣━━ bib/
┗━━ out/

With all build files in the out/ folder and not at project root. This would make giting more convenient as well, with a simple /out/* line added to the .gitignorefile.

Issue

Using the information found in this topic (most complete topic on the subject, I believe), I came up running the following command[2]:

lualatex -synctex=1 --output-directory=./out -interaction=nonstopmode ./biblio_test.tex

which works well and creates a bunch of things in the out folder, followed by:

bibtex ./out/biblio_test.aux

However bibtex fails with the following error:

I couldn't open database file biblio_test-blx.bib
---line 3 of file out/biblio_test.aux
 : \bibdata{biblio_test-blx
 :                         ,bib/my_bib}

Which is caused to the very content of the lualatex-generated biblio_test.aux file:

\bibdata{biblio_test-blx,bib/my_bib}

Which should instead be: \bibdata{out/biblio_test-blx,bib/my_bib}.

Temporary fix

I can think of two ways to fix this issue for now (tested, working):

• move the biblio_text-blx.bib to the root folder.
• manually add the out/ relative path to the biblio_test.aux file.

However none of these are solutions since those files will change at each compilation.

Questions

• Is this a bug of the lualatex command? Am I doing something terribly wrong here?
• How can I solve this ?

Additional Information

• I also tried with the pdflatex command (pdflatex -synctex=1 -output-directory=./out -interaction=nonstopmode ./"biblio_test".tex, only one dash for the -output-directory argument).
• My LaTeX distribution is TeX Live 2020, running under Mac OS Catalina (10.15.6).
• I can also think of dirty-but-working solutions such as defining an alias for my bibtex command doing something like (bash draft) mv out/biblio_test.aux ./biblio_test.aux && bibtek ./biblio_test.aux && mv ./biblio_test.bbl out/biblio_test.bbl && rm ./biblio_test.aux, but I would prefer not to.

---

[1]: The original question was unanswered and labelled as abandoned. Following the discussion posted on meta on this topic, I just re-asked an improved version of the original question, and answered it.
[2]: Important: be sure to create an out folder first with mkdir out or the like if using the proposed minimal working example.

Answer 1

Well that is a very good example why using --output-directory is problematic (and why I never use that).

The internal bib-file written by biblatex is, as you requested, written to the out directory, but the internal commands of biblatex don't know that. So here the out/ is missing:

\def\blx@auxinit#1{%
  \blx@auxwrite\blx@aux
    {\def\do##1{,\blx@stripbib{##1}}}
    {\ifx\blx@aux\@mainaux
     \else
       \blx@msg@aux
     \fi
     \string\bibstyle{biblatex}\blx@nl
     \string\bibdata{%
       \blx@ctrlfile\blxauxsuffix %<--- should be out/\blx@ctrlfile\blxauxsuffix
       \ifx#1\@empty
       \else
         \dolistloop#1%
       \fi}\blx@nl
     \string\citation{biblatex-control}}}

With biber there is no problem as it doesn't require a special bib to work.

You can save yourself a lot of debugging time (and save packages author from the trouble to have to consider such setups and add options for this) by simply leaving the files where the tools and packages expect them to be.

Answer 2

NOT TESTED May be you should run BibTex with out begin the current directory, ie in TeXStudio use this template

cd out && bibtex %.aux

On top of that create in the root directory a symlink biblio_test-blx.bbl pointing at out/biblio_test-blx.bbl

Answer 3

I have no absolute answer as to the intricate details of what is going on behind the scenes, but from what I understood, the issue was with bibtex. Indeed, I did not find a way to tell bibtexto do its whole stuff inside the out directory. I could only tell it to look for the .aux file inside the out directory with the command:

bibtex ./out/biblio_test.aux

But it always search the biblio_test-blx in the root folder and not in the out directory.

Solution: use biber

On the contrary, biber does provide two options for specifying the input / output directory (complete option list available through biber --help):

• --input-directory [directory]: .bcf and data files will be looked for first in the directory. See the biber PDF documentation for the other possibilities and how this interacts with the "--output-directory" option.
• --output-directory [directory]: Output files (including log files) are output to directory instead of the current directory. Input files are also looked for in directory before current directory unless "--input-directory" is also specified in which case input files are only looked for in the directory specified by "--input-directory".

I then replaced in TeXStudio the bibliographic tool command line:

bibtex out/%.aux

by

biber --output-directory out %

This works like a charm and allows one to use the advanced functionalities of the biber backend. For more consideration on the differences between bibtex and biber as well as other technical considerations, one may be interested in reading the following: bibtex vs. biber and biblatex vs. natbib

---

Do not hesitate to improve this answer or signal appropriate remarks in the comments, I'll upgrade my answer accordingly. I know this is only a workaround and does not really address the initial bibtex issue. But it is the best I came up with so far.

Answer 4

For those who still want to use -output-directory and bibtex at the same time and avoid the <report_name>-blx.bib problem... here it is a workaround:

In your makefile:

DOCNAME=main
OUTDIR=out
BLX=main-blx.bib

.PHONY: clean

all: pdf

pdf: | outdir
    pdflatex --shell-escape -output-directory=$(OUTDIR) $(DOCNAME).tex
    MOVE .\$(OUTDIR)\$(BLX) .
    bibtex $(OUTDIR)/$(DOCNAME)
    del $(BLX)
    pdflatex --shell-escape -output-directory=$(OUTDIR) $(DOCNAME).tex
    pdflatex --shell-escape -output-directory=$(OUTDIR) $(DOCNAME).tex
clean:
    rmdir $(OUTDIR)

This way works perfectly and since it is just one time used by bibtex, it's okay to remove it... it will be generated in the next make command.