biblatex: \nocite{*} causes TeX capacity exceeded

Question

I am using \nocite{*} to create a typeset bibliography of the complete contents of all my .bib files. Under Ubuntu 18 (TexLive 2017), this worked perfectly and gave a 200 page document. Now Ubuntu 20 (TexLive 2019), I get "TeX capacity exceeded" when the document size goes over about 150 pages. The exact error message is:

Overfull \hbox (5.53261pt too wide) in paragraph at lines 182--182
[][]\T1/zgmx/m/n/10 Laur, Wolf-gang (1954), `Sli-esthorp, Schleswig, Hedeby und
 Had-deby. Die Na-men von Schleswig-
[125]
Runaway argument?
Lennard:Lawshall|Lennard:satellite|Lennard_1959|Lepelley:Normandiedia\ETC.
! TeX capacity exceeded, sorry [main memory size=5000000].
\blx@segm@0@0 ...nshire|Padel:Parochia|Padel:chang
                                                  e|commodity|JEPNS-39-Padel...
l.182 ...ibliography[heading=bibliography-heading]
                                                  
!  ==> Fatal error occurred, no output PDF file produced!

The files below give a MWE. The python script generates a dummy .bib file with random entries.

make_bibtex_file.py:

# python3 make_bibtex_file.py >| foo.bib && texclean && latexmk -pdf bibliography_all

from random import seed,random,randrange,choices,choice

book='''@book{%s,
  author=   {%s},
  title=    {%s},
  publisher={%s},
  address=  {%s},
  year=     {%d},
}
'''

article='''@article{%s,
  author= {%s},
  title=  {%s},
  journal={%s},
  volume= {%d},
  pages=  {%d--%d},
  year=   {%d},
}
'''

def go(na,nb):
  alphabet='abcdefghijklmonpqrstuvwxyz'
  for i in range(na):
    k=''.join(choices(alphabet,k=10)).title()
    a=choice(alphabet).upper()+'. '+(('',choice(alphabet).upper()+'. ')[random()<0.7])+''.join(choices(alphabet,k=8)).title()
    t=''.join(choices(alphabet,k=randrange(3,12))).title()+' '+''.join(choices(alphabet,k=randrange(3,12))).title()+' '+''.join(choices(alphabet,k=randrange(3,12))).title()
    j=''.join(choices(alphabet,k=randrange(3,12))).title()+' '+''.join(choices(alphabet,k=randrange(3,12))).title()
    v=randrange(1,200)
    p=''.join(choices(alphabet,k=randrange(5,12))).title()
    l=''.join(choices(alphabet,k=randrange(6,12))).title()
    p0=randrange(1,200)
    p1=p0+randrange(1,40)
    y=randrange(1900,2023)
    print(article%(k,a,t,j,v,p0,p1,y))
  for i in range(nb):
    k=''.join(choices(alphabet,k=10)).title()
    a=choice(alphabet).upper()+'. '+(('',choice(alphabet).upper()+'. ')[random()<0.7])+''.join(choices(alphabet,k=8)).title()
    t=''.join(choices(alphabet,k=randrange(3,12))).title()+' '+''.join(choices(alphabet,k=randrange(3,12))).title()+' '+''.join(choices(alphabet,k=randrange(3,12))).title()
    p=''.join(choices(alphabet,k=randrange(5,12))).title()
    l=''.join(choices(alphabet,k=randrange(6,12))).title()
    y=randrange(1900,2023)
    print(book%(k,a,t,p,l,y))

if __name__=='__main__':
  seed(1)
  go(na=3000,nb=3000) # 3000,3000 ok => 123 pages
  #go(na=4000,nb=3000) # not ok
  #go(na=3500,nb=3000) # not ok
  #go(na=3400,nb=3000) # not ok
  #go(na=3300,nb=3000) # not ok
  #go(na=4000,nb=4000) # not ok => (./bibliography_all.bbl Runaway text?  Szurrycplj|Dpopvymmif|Khzggpogfj|Iazmfsnmey|Wpczdiuelp|Yyktvieuhz|Mku\ETC.  ! TeX capacity exceeded, sorry [main memory size=5000000].  \blx@dlist@entry@0@nyt/global//global/global ...kj ggsjyc|Elfwxhwise|Lhznuyxn...  l.206098     \endentry

bibliography_all.tex:

\documentclass[10pt,a4paper]{article}
\usepackage[T1]{fontenc}
\usepackage[german,american,english]{babel}
\usepackage[babel]{csquotes}
\usepackage{time}
\usepackage{garamondx}
\usepackage[lmargin=60mm,rmargin=15mm,tmargin=20mm,bmargin=20mm]{geometry}
\usepackage{ragged2e}
\usepackage{showkeys}
\usepackage[style=authoryear,sorting=nyt,hyperref,dashed=true,backend=biber,bibencoding=utf8,texencoding=ascii]{biblatex}

\bibliography{foo}

\usepackage[colorlinks,linkcolor=blue,anchorcolor=blue,citecolor=blue,urlcolor=blue,pdfauthor={Keith Briggs},pdfmenubar=false,pdffitwindow=true,pdfwindowui=false,pdftitle={A big bibliography},plainpages=false]{hyperref}

\title{A big bibliography}
\author{Keith Briggs}
\def\Today{\number\year\space
 \ifcase\month\or January\or February\or March\or April\or May\or June\or
   July\or August\or September\or October\or November\or December\fi
     \space\number\day
}
\date{Last revised \Today\ \now}

\begin{document}
\renewcommand{\baselinestretch}{1.1}
\maketitle
\nocite{*}
\printbibliography
\end{document}

Answer 1

Just to get a baseline I ran the following documents with Overleaf's TeX Live 2017 and TeX Live 2019.

Document 1

\documentclass{article}
\usepackage[utf8]{inputenc}

\begin{document}
Lorem
\end{document}

Document 2

\documentclass{article}
\usepackage[utf8]{inputenc}

\usepackage[style=authoryear]{biblatex}
\addbibresource{biblatex-examples.bib}

\begin{document}
\section{Introduction}
Lorem
\autocite{sigfridsson}
\nocite{*}
\printbibliography
\end{document}

The following table shows how many words of memory out of 5000000 TeX used in each document with the given TeX Live versions.

Document	2017	2019	difference
1	56297	62012	~10%
2	555443	594316	~7%

I'm afraid it is how it is: New features sometimes come with more complex or new implementations that need more memory.

You can try to compile your document with LuaTeX, which has dynamic memory allocation and is usually much better with memory-heavy documents. (There is also How to expand TeX's "main memory size"? (pgfplots memory overload) suggesting a few methods to increase TeX's memory, but there is still a hard-coded limit and Ulrike Fischer's comment suggests that is still not enough).