Prevent babel shorthands from interfering with catcode changes

Question

I want to write an XML-like key-value-parser for LaTeX. That is, kv-pairs are separated by spaces, and values are separated from keys by = and encapsulated by double qoutes. The following minimal example does that:

\documentclass[english]{book}
\usepackage{babel}

\makeatletter
\def\tp@parse@attributes#1\@nil{%
  \if!#1!\else\@tp@parse@attributes#1 ="" =""\@nil\fi}

\def\@tp@parse@attributes#1="#2" #3=""\@nil{%
  \edef\@argi{#1}\edef\@argiii{#3}%
  \ifx\@argi\@empty\else%
    \ifx\@argi\space\else%
      \expandafter\def\csname tp@attr@#1\endcsname{#2}%
      \ifx\@argiii\@empty\else
        \ifx\@argiii\space\else
          \def\@tempa{="" }%
          \ifx\@argiii\@tempa\else%
            \expandafter\@tp@parse@attributes #3=""\@nil%
          \fi\fi\fi\fi\fi}%

\def\test{\@ifnextchar [\@test{\@test[class="default"]}}%]
\def\@test[#1]#2{%
  \tp@parse@attributes#1\@nil
  Class: {\ttfamily\tp@attr@class}\space
  Content: {\sffamily #2}\par
  \let\tp@attr@class\relax}% reset class for next instance
\parindent\z@
\makeatother

\begin{document}
1st test: \test{Content}
2nd test: \test[class="foobar"]{Content}
\end{document}

This works as long as the document language is english.

For german texts, the same code throws an error ! Argument of \language@active@arg" has an extra }. I figured that babel shorthands are to blame, so I played around with the catcodes and came up with the following code (… means that the code is the same as in the MWE above):

\documentclass[german]{book}
\usepackage{babel}

\makeatletter
\begingroup
\catcode`\"=\active
\gdef\tp@parse@attributes#1\@nil{…}
\gdef\@tp@parse@attributes#1="#2" #3=""\@nil{…}%
\gdef\test{…}%
\gdef\@test[#1]#2{…}
\endgroup

\parindent\z@
\makeatother

\begin{document}
…
\end{document}

Unfortunately, this option does not work, when the (only) language is english (! Undefined control sequence.^^J<argument> ="). However, it does work when both languages are loaded (\documentclass[english,german]{book})

If the language turkish is added (or single language), none of those options work.

I also attempted to locally switch off babel shorthands using \shorthandoff{"=}, but since I do not know what languages are actually loaded, I cannot anticipate which characters are active shorthands and which are not.

My question is: How must I alter my code so it works with all languages, and without globally deactivating babel shorthands?

Answer 1

The most common case of having to handle babel active characters in a keyval list is that for Turkish, where = and , are active. The same techniques can be applied to " as for = and , - it is simply a question of adding an additional step.

You have basically two possible approaches, which are really just variations on the same idea: we need to only look for delimited arguments with the correct category codes. So you either

• Parse over the raw list first and replace 'active' =, ,, " by 'other' tokens of the same catcode, then do your keyval parsing only using 'other' catcodes

• Parse the raw list using a two-step approach where you first look for 'active' then for 'other' versions of each token

There are some performance issues to think about in determining which to do, for which I'd suggest looking at the expkv implementation. I think at a conceptual level it's probably easiest to go with first replacing all, then doing a simple search. As a proof-of-concept, I'll demo that by using expl3's search-and-replace function - this is not the most efficient way to do it, but it will show one possible approach

\documentclass[german]{book}
\usepackage{babel}
\ExplSyntaxOn
\cs_new_eq:NN \ReplaceAll \tl_replace_all:Nnn
\ExplSyntaxOff
\makeatletter
\begingroup
  \catcode`\"=\active
  \xdef\tp@parse@attributes#1\@nil{%
    \def\noexpand\@tempa{#1}%
    \ReplaceAll\noexpand\@tempa{\noexpand"}{\string"}%
    \noexpand\expandafter\noexpand\tp@parse@attributes@aux
      \noexpand\expandafter{\noexpand\@tempa}%
  }
\endgroup
\def\tp@parse@attributes@aux#1{%
  \if!#1!\else\@tp@parse@attributes#1 ="" =""\@nil\fi}

\def\@tp@parse@attributes#1="#2" #3=""\@nil{%
  \edef\@argi{#1}\edef\@argiii{#3}%
  \ifx\@argi\@empty\else%
    \ifx\@argi\space\else%
      \expandafter\def\csname tp@attr@#1\endcsname{#2}%
      \ifx\@argiii\@empty\else
        \ifx\@argiii\space\else
          \def\@tempa{="" }%
          \ifx\@argiii\@tempa\else%
            \expandafter\@tp@parse@attributes #3=""\@nil%
          \fi\fi\fi\fi\fi}%

\def\test{\@ifnextchar [\@test{\@test[class="default"]}}%]
\def\@test[#1]#2{%
  \tp@parse@attributes#1\@nil
  Class: {\ttfamily\tp@attr@class}\space
  Content: {\sffamily #2}\par
  \let\tp@attr@class\relax}% reset class for next instance
\parindent\z@
\makeatother

\begin{document}
1st test: \test{Content}
2nd test: \test[class="foobar"]{Content}
\end{document}