Prevent babel shorthands from interfering with catcode changes

I want to write an XML-like key-value-parser for LaTeX. That is, kv-pairs are separated by spaces, and values are separated from keys by = and encapsulated by double qoutes. The following minimal example does that:

\documentclass[english]{book}
\usepackage{babel}

\makeatletter
\def\tp@parse@attributes#1\@nil{%
\if!#1!\else\@tp@parse@attributes#1 ="" =""\@nil\fi}

\def\@tp@parse@attributes#1="#2" #3=""\@nil{%
\edef\@argi{#1}\edef\@argiii{#3}%
\ifx\@argi\@empty\else%
\ifx\@argi\space\else%
\expandafter\def\csname tp@attr@#1\endcsname{#2}%
\ifx\@argiii\@empty\else
\ifx\@argiii\space\else
\def\@tempa{="" }%
\ifx\@argiii\@tempa\else%
\expandafter\@tp@parse@attributes #3=""\@nil%
\fi\fi\fi\fi\fi}%

\def\test{\@ifnextchar [\@test{\@test[class="default"]}}%]
\def\@test[#1]#2{%
\tp@parse@attributes#1\@nil
Class: {\ttfamily\tp@attr@class}\space
Content: {\sffamily #2}\par
\let\tp@attr@class\relax}% reset class for next instance
\parindent\z@
\makeatother

\begin{document}
1st test: \test{Content}
2nd test: \test[class="foobar"]{Content}
\end{document}


This works as long as the document language is english.

For german texts, the same code throws an error ! Argument of \language@active@arg" has an extra }. I figured that babel shorthands are to blame, so I played around with the catcodes and came up with the following code (… means that the code is the same as in the MWE above):

\documentclass[german]{book}
\usepackage{babel}

\makeatletter
\begingroup
\catcode\"=\active
\gdef\tp@parse@attributes#1\@nil{…}
\gdef\@tp@parse@attributes#1="#2" #3=""\@nil{…}%
\gdef\test{…}%
\gdef\@test[#1]#2{…}
\endgroup

\parindent\z@
\makeatother

\begin{document}
…
\end{document}


Unfortunately, this option does not work, when the (only) language is english (! Undefined control sequence.^^J<argument> ="). However, it does work when both languages are loaded (\documentclass[english,german]{book})

If the language turkish is added (or single language), none of those options work.

I also attempted to locally switch off babel shorthands using \shorthandoff{"=}, but since I do not know what languages are actually loaded, I cannot anticipate which characters are active shorthands and which are not.

My question is: How must I alter my code so it works with all languages, and without globally deactivating babel shorthands?

• I would be tempted to make " always active. expanding to a normal " then you can just use the active " in your delimited arguments May 10 at 7:38
• I'm not sure if it is a good idea to create a new user key-val input syntax. It would feel awkward to have to use in one document sometimes class="default" color="red" and sometimes class=default, color=red. May 10 at 7:40
• @DavidCarlisle this doesn't seem to work for neither english nor for turkish alone. I assume that " is not the only character that is altered by babel for turkish, otherwise the german fix should work (right?). Besides, i don't want to risk destroying any other packages by globally altering catcodes like that. May 10 at 7:43
• @UlrikeFischer that is a fair point, however not my question. Whether i actually adopt this syntax or not is still somewhat open. The background is that some css values use , as separators and i fear that such values may interfer with the parser of some KV-packages. Besides, the answer would help me greatly to understand how catcodes work and what can be done to cope with catcode conflicts like that. ("Es geht um's Prinzip") May 10 at 7:51

The most common case of having to handle babel active characters in a keyval list is that for Turkish, where = and , are active. The same techniques can be applied to " as for = and , - it is simply a question of adding an additional step.

You have basically two possible approaches, which are really just variations on the same idea: we need to only look for delimited arguments with the correct category codes. So you either

• Parse over the raw list first and replace 'active' =, ,, " by 'other' tokens of the same catcode, then do your keyval parsing only using 'other' catcodes

• Parse the raw list using a two-step approach where you first look for 'active' then for 'other' versions of each token

There are some performance issues to think about in determining which to do, for which I'd suggest looking at the expkv implementation. I think at a conceptual level it's probably easiest to go with first replacing all, then doing a simple search. As a proof-of-concept, I'll demo that by using expl3's search-and-replace function - this is not the most efficient way to do it, but it will show one possible approach

\documentclass[german]{book}
\usepackage{babel}
\ExplSyntaxOn
\cs_new_eq:NN \ReplaceAll \tl_replace_all:Nnn
\ExplSyntaxOff
\makeatletter
\begingroup
\catcode\"=\active
\xdef\tp@parse@attributes#1\@nil{%
\def\noexpand\@tempa{#1}%
\ReplaceAll\noexpand\@tempa{\noexpand"}{\string"}%
\noexpand\expandafter\noexpand\tp@parse@attributes@aux
\noexpand\expandafter{\noexpand\@tempa}%
}
\endgroup
\def\tp@parse@attributes@aux#1{%
\if!#1!\else\@tp@parse@attributes#1 ="" =""\@nil\fi}

\def\@tp@parse@attributes#1="#2" #3=""\@nil{%
\edef\@argi{#1}\edef\@argiii{#3}%
\ifx\@argi\@empty\else%
\ifx\@argi\space\else%
\expandafter\def\csname tp@attr@#1\endcsname{#2}%
\ifx\@argiii\@empty\else
\ifx\@argiii\space\else
\def\@tempa{="" }%
\ifx\@argiii\@tempa\else%
\expandafter\@tp@parse@attributes #3=""\@nil%
\fi\fi\fi\fi\fi}%

\def\test{\@ifnextchar [\@test{\@test[class="default"]}}%]
\def\@test[#1]#2{%
\tp@parse@attributes#1\@nil
Class: {\ttfamily\tp@attr@class}\space
Content: {\sffamily #2}\par
\let\tp@attr@class\relax}% reset class for next instance
\parindent\z@
\makeatother

\begin{document}
1st test: \test{Content}
2nd test: \test[class="foobar"]{Content}
\end{document}