Loop through Entire Bib Database

Question

I would like to loop through all entries in the BibLaTeX database and generate a \csxdef based on the each key. The pseudo code for what I desire is:

\foreach \EntryKey in \ListOfAllBibliography {%
    \edef\UrlKeyValue{\citefield{\EntryKey}{url}}%
    \StrBehind*{\UrlKeyValue}{//}[\UrlKeyValueExtracted]%
    \IfStrEq{\UrlKeyValue}{}{}{%
        \csxdef{Bib \UrlKeyValueExtracted}{\EntryKey}%
    }%
}%

For this particular case, this loop should be equivalent to

\csxdef{Bib books.google.com/books?id=hEYuAQAAIAAJ}{knuth1984texbook}%
\csxdef{Bib books.google.com/books?id=54A3MuBzIrEC}{goossens1994latex}%

The expected output of the MWE when this works is:

References:

• One option would be to do this in python as per Converting bib database into csv, but would prefer to do it in LaTeX.

Code:

\begin{filecontents}{mybib.bib}
@book{knuth1984texbook,
  title={The texbook},
  author={Knuth, D.E. and Knuth, A.D. and Bibby, D. and American Mathematical Society and Addison-Wesley Publishing Company and Addison-Wesley},
  isbn={9780201134483},
  lccn={85030845},
  series={Computers \& typesetting},
  url={https://books.google.com/books?id=hEYuAQAAIAAJ},
  year={1984},
  publisher={Addison-Wesley},
  myFieldA={Useful Book},
  myFieldB={on Shelf 4},
}
@book{goossens1994latex,
  title={The LaTeX Companion},
  author={Goossens, M. and Mittelbach, F. and Samarin, A.},
  isbn={9780201541991},
  lccn={lc93023150},
  series={Addison-Wesley series on tools and techniques for computer typesetting},
  url={https://books.google.com/books?id=54A3MuBzIrEC},
  year={1994},
  publisher={Addison-Wesley},
  myFieldA={Also Useful Book},
  myFieldB={on Shelf 5},
}

\end{filecontents}

\documentclass{article}
\usepackage{xstring}% used only in pseudo code for this MWE.

\usepackage{pgffor}
\usepackage{etoolbox}

\usepackage{biblatex}
\addbibresource{mybib.bib}

\begin{document}

%% Pseudo code for desired looping:
%    \foreach \EntryKey in \ListOfAllBibliography {%
%        \edef\UrlKeyValue{\citefield{\EntryKey}{url}}%
%        \StrBehind*{\UrlKeyValue}{//}[\UrlKeyValueExtracted]%
%        \IfStrEq{\UrlKeyValue}{}{}{% Skip if the url= key us missing
%            \csxdef{Bib \UrlKeyValueExtracted}{\EntryKey}%
%        }%
%    }%

\foreach \x in {hEYuAQAAIAAJ, 54A3MuBzIrEC} {%
    \ifcsdef{Bib books.google.com/books?id=\x}{%
        PASS: Found (\citefield{\csuse{Bib books.google.com/books?id=\x}}{title}).
    }{%
        FAIL: Missing csdef for \x.
    }%
    \par
}

\end{document}

Answer 1

From within biblatex we can never loop over all entries in the .bib database. We can only loop over all entries that are in the .bbl file. If you manually cited all entries or used \nocite{*} there should usually not be that much of a difference, but if you use some more advanced feature like entry cloning or crossref, you may end up with additional "cloned" entries that are not reflected in the original database.

The best way to loop over all available entries is probably \AtDataInput. This hook is executed whenever an entry is read from the .bbl file (at the very end of the read-in). At this points fields are available expandable as command sequences \abx@field@<field name> (so you don't have to fiddle with \citefield and friends which are not expandable and thus no good in the example anyway).

\documentclass{article}
\usepackage{xstring}
\usepackage{pgffor}
\usepackage{biblatex}

\makeatletter
\AtDataInput{%
  \ifundef\abx@field@url
    {}
    {\edef\UrlKeyValue{\abx@field@url}%
     \StrBehind*{\UrlKeyValue}{//}[\UrlKeyValueExtracted]%
     \csxdef{Bib \UrlKeyValueExtracted}{\abx@field@entrykey}}}
\makeatother

\begin{filecontents}{\jobname.bib}
@book{knuth1984texbook,
  title={The texbook},
  author={Knuth, D.E. and Knuth, A.D. and Bibby, D.},
  isbn={9780201134483},
  lccn={85030845},
  series={Computers \& typesetting},
  url={https://books.google.com/books?id=hEYuAQAAIAAJ},
  year={1984},
  publisher={Addison-Wesley},
  myFieldA={Useful Book},
  myFieldB={on Shelf 4},
}
@book{goossens1994latex,
  title={The LaTeX Companion},
  author={Goossens, M. and Mittelbach, F. and Samarin, A.},
  isbn={9780201541991},
  lccn={lc93023150},
  series={Addison-Wesley series on tools and techniques for computer typesetting},
  url={https://books.google.com/books?id=54A3MuBzIrEC},
  year={1994},
  publisher={Addison-Wesley},
  %pages={11-15},
  myFieldA={Also Useful Book},
  myFieldB={on Shelf 5},
}

\end{filecontents}
\addbibresource{\jobname.bib}

\begin{document}
\nocite{*}

\foreach \x in {hEYuAQAAIAAJ, 54A3MuBzIrEC} {%
    \ifcsdef{Bib books.google.com/books?id=\x}{%
        PASS: Found (\citefield{\csuse{Bib books.google.com/books?id=\x}}{title}).
    }{%
        FAIL: Missing csdef for \x.
    }%
    \par
}
\end{document}

Answer 2

You may know about my usebib package, but I also have an experimental expl3 version of it. Your question prompted me to add a feature, namely a sequence where the citation keys are stored.

How does usebib3 work? For each entry in the bib file(s) we load it sets up a property list containing the entry's fields. Then we can access each field by retrieving it from the property list.

In your case, you can make a new property list containing as keys the url fields and as values the citation keys they belong to.

usebib3.sty (version 0.3)

\RequirePackage{expl3,xparse,url}
\ProvidesExplPackage{usebib3}{2022/05/06}{v. 0.3}{Use fields read in bib files}

\prop_new:N \l__usebib_temp_prop
\prop_new:N \g__usebib_temp_prop
\seq_new:N \g_usebib_discard_seq
\seq_new:N \g_usebib_keys_seq

\NewDocumentCommand{\bibinput}{m}
 {
  \clist_map_function:nN { #1 } \usebib_main:n
 }
\NewExpandableDocumentCommand{\usebibentry}{mm}
 {
  \prop_item:cf { g_usebib_entry_#1_prop } { \str_lower_case:n { #2 } }
 }
\NewExpandableDocumentCommand{\usebibentryurl}{O{|}m}
 {
  \__usebib_url:nf { #1 } { \prop_item:cn { g_usebib_entry_#2_prop } { url } }
 }
\NewDocumentCommand{\newbibfield}{m}{}

\@ifpackageloaded{hyperref}
 {
  \cs_new_protected:Nn \__usebib_url:nn
   {
    \url{ #2 }
   }
 }
 {
  \cs_new_protected:Nn \__usebib_url:nn
   {
    \tl_rescan:nn { \url #1 #2 #1 }
   }
 }
\cs_generate_variant:Nn \__usebib_url:nn { nf }
\cs_generate_variant:Nn \prop_item:Nn { cf }

\NewDocumentCommand{\newbibignore}{m}
 {
  \seq_gput_right:Nx \g_usebib_discard_seq { \str_lower_case:n { #1 } }
 }

\cs_new_protected:Nn \usebib_main:n
 {
  \group_begin:
  \char_set_active_eq:NN @ \usebib_read_entry:w
  \char_set_catcode_active:n { `@ }
  \file_input:n { #1.bib }
  \group_end:
 }

\cs_new_protected:Npn \usebib_read_entry:w #1#
 {
  \seq_if_in:NxTF \g_usebib_discard_seq { \str_lower_case:n { #1 } }
   {
    \use_none:n
   }
   {
    \__usebib_read_entry:w
   }
 }

\cs_new_protected:Npn \__usebib_read_entry:w
 {
  \group_begin:
  \char_set_catcode_other:n { `@ }  % may appear in text
  \char_set_catcode_other:n { `\% } % can be used in URLs
  \char_set_catcode_other:n { `\# } % can be used in fields
  \__usebib_read_entry:n
 }

\cs_new_protected:Nn \__usebib_read_entry:n
 {
  \prop_gset_from_keyval:Nn \g__usebib_temp_prop { USEBIBREADKEY=#1 }
  \group_end:
  \__usebib_save_entry:
 }

\cs_new_protected:Nn \__usebib_save_entry:
 {
  \prop_clear:N \l__usebib_temp_prop
  \prop_map_inline:Nn \g__usebib_temp_prop
   {
    \str_if_eq:nnTF { ##1 } { USEBIBREADKEY }
     {
      \seq_gput_right:Nn \g_usebib_keys_seq { ##2 }
     }
     {
      \prop_put:Nxn \l__usebib_temp_prop { \str_lower_case:n { ##1 } } { ##2 }
     }
   }
  \prop_new:c { g_usebib_entry_ \prop_item:Nn \g__usebib_temp_prop { USEBIBREADKEY } _prop }
  \prop_gset_eq:cN
   { g_usebib_entry_ \prop_item:Nn \g__usebib_temp_prop { USEBIBREADKEY } _prop }
   \l__usebib_temp_prop
 }
\cs_generate_variant:Nn \prop_put:Nnn { Nx }

Test file using your bib file renamed petergrill.bib (not shown here)

\documentclass{article}
\usepackage{usebib3}

\bibinput{petergrill}

\ExplSyntaxOn

\prop_new:N \l_petergrill_url_prop

%% save the url fields
\seq_map_inline:Nn \g_usebib_keys_seq
 {
  \prop_put:Nxn \l_petergrill_url_prop { \usebibentry { #1 } { url } } { #1 }
 }

\NewDocumentCommand{\checkids}{mm}
 {% #1 = prefix, #2 = list
  \clist_map_inline:nn { #2 }
   {
    \prop_if_in:NnTF \l_petergrill_url_prop { #1##1 }
     {
      PASS:~Found~(\usebibentry{\prop_item:Nn \l_petergrill_url_prop { #1##1 }}{title}) \par
     }
     {
      FAIL:~Missing~##1
     }
   }
 }

\ExplSyntaxOff

\begin{document}

\checkids{https://books.google.com/books?id=}{hEYuAQAAIAAJ, 54A3MuBzIrEC}

\end{document}

The \seq_map_inline:Nn part populates the property list mentioned at the beginning and \checkids looks whether the items are there, printing the title field in this case.