# Process a large glossary?

I have a 1.1MB file with 20k acronyms (gene names). I don't use all of the acronyms in my text, but I still get the error message below, when I try to compile. How can I avoid this?

(./tex/glossary.tex (./genes.tex
Runaway definition?
->,{A1BG},{A1CF},{A2M},{A2ML1},{A3GALT2},{A4GALT},{A4GNT},{AAAS},{AAC\ETC.
! TeX capacity exceeded, sorry [main memory size=5000000].
\glolist@acronym ...UFB2},{NDUFB3},{NDUFB4},{NDUFB
5},{NDUFB6},{NDUFB7},{NDUF...
l.10470 ...ssed, developmentally down-regulated 1}

• For me it stops at \newacronym{PIGK}{PIGK}{phosphatidylinositol glycan anchor biosynthesis class K}, having some higher main memory size, but it takes very long to get there – user31729 Feb 28 '18 at 22:50
• Perhaps bib2gls is an option for you – user31729 Feb 28 '18 at 22:56
• You could perhaps use lualatex which uses dynamic memory allocation but it's very big (my texlive 2017 version finally dies with ! TeX capacity exceeded, sorry [number of strings=494451]., I would instead just get latex to write out which acronyms are used, then use perl to make a subset list of definitions to input. – David Carlisle Feb 28 '18 at 22:57
• @tommy.carstensen if you posted a test file it would be easier to suggest how to get there – David Carlisle Feb 28 '18 at 22:59
• @tommy.carstensen: I would rather split the huge file into more 'logical' files, e.g. a file containing some frequently used genes ... – user31729 Feb 28 '18 at 23:09

A quick and (dirty) way (most likely I have reinvented some glossaries - wheel ;-)

The idea is following:

• Redefine \gls (and other similar macros), putting the used one into a list of used gls entries
• Store those values to a separate file, say \jobname.used
• Read in those values before \loadglsentries and redefine \newacronym to use only those entries that match the ones that have been stored, kick the other ones.

\documentclass{article}

\usepackage{xparse}
\usepackage{letltxmacro}
\usepackage{glossaries}

\makeglossaries

\LetLtxMacro\newacronymdefault\newacronym
\LetLtxMacro\glsorigdefault\gls

\newwrite\usedentries
\AtBeginDocument{
\immediate\openout\usedentries=\jobname.used
}

\newif\ifshrinkglossaryentries

\shrinkglossaryentriestrue

\ExplSyntaxOn

\newcommand{\listofused}[1]{%
}

\seq_new:N \gls_used_seq

\ifshrinkglossaryentries

\RenewDocumentCommand{\newacronym}{O{}+m+m+m}{%
\newacronymdefault[#1]{#2}{#3}{#4}%
}%
}
\fi
\ExplSyntaxOff

\InputIfFileExists{\jobname.used}{}{}

\ExplSyntaxOn
\RenewDocumentCommand{\gls}{+O{}+m}{%
\glsorigdefault[#1]{#2}%
\seq_gput_right:Nn \gls_used_seq {#2}
}

\makeatletter
\AtEndDocument{%
\immediate\write\usedentries{%
\string\listofused{\seq_use:Nnnn \gls_used_seq { ,}{,}{}}
}
\immediate\closeout\usedentries
}
\makeatother

\ExplSyntaxOff

\begin{document}

\gls{PIGH}

\gls{ZZZ3}

\printglossaries

\end{document}

• Oh man, I feel so stupid, when you make it look so easy :D Thanks!!! – tommy.carstensen Feb 28 '18 at 23:38
• @tommy.carstensen: You're welcome. I'll try to improve it later on – user31729 Feb 28 '18 at 23:39

The glossaries performance page compares the results of document builds for some files containing 1000 entry definitions, using a number of different methods provided by the base glossaries package and the glossaries-extra extension package. In particular, check the Alphabetical Order (Subset) section. The files in these tests aren't as large as with your document, but it's possible to see which methods perform better for large datasets.

The best method for a large set of entries where only a subset is required in the document is to use glossaries-extra with bib2gls. This has a considerable saving in resources as only those entries that are actually needed in the document require storage within TeX. (Naturally, bib2gls will need enough memory, but it's a Java application so it has better memory management than TeX.)

• Thanks! I would probably have gone for bib2gls, if I had to do it again, instead of pruning the glossary of human gene IDs. – tommy.carstensen Mar 2 '18 at 16:54
• @tommy.carstensen You might be able to convert your .tex file to .bib format with convertgls2bib. (One drawback is that bib2gls is quite new, so there may be bugs that haven't been found yet.) – Nicola Talbot Mar 2 '18 at 17:03