Replace chars with xstring. Catcode problem

Question

I try to do something like this (minimal example):

\documentclass[]{article}
\usepackage{mathtools}
\usepackage{xstring}

\newcommand{\foo}[1]{
    \StrSubstitute{#1}{:}{\string&}[\fooo]
    \StrSubstitute{\fooo}{-}{\string\\}[\fooo]
    \tokenize{\fooo}{\fooo}
    \fooo}


\begin{document}


\[\begin{matrix}
    \foo{1:2-2:3}
\end{matrix}\]
\end{document}

But I it isn't working, for some reason this is working:

\documentclass[]{article}
\usepackage{mathtools}
\usepackage{xstring}

\begin{document}

\StrSubstitute{1:2-2:3}{:}{\string&}[\fooo]
\StrSubstitute{\fooo}{-}{\string\\}[\fooo]
\tokenize{\fooo}{\fooo}

\[\begin{matrix}
    \fooo
\end{matrix}\]
\end{document}

Any Idea on this? I think this is related to catcodes...

Maybe someone also has an explanation for this behavior?

Answer 1

The reason your code doesn't work is that the matrix environment (or rather, the underlying \halign TeX primitive) sees & while xstring is manipulating it (more precisely, while performing the \tokenize step). TeX thinks that this is the end of the cell, inserts the end-of-cell material, including the internal primitive \endtemplate, which makes the xstring macros choke. If you want to see this in action, add \tracingall before \tokenize, and get a confusing output.

So... how do we hide & from TeX's inquisitive eyes? It turns out that if TeX sees & (or rather, a token with catcode 4, alignment tab) when it thinks that it is within braces, the & token is left as it is, and not converted to the end of a cell. Thus for instance \def\foo{&} will not make TeX choke when appearing in an alignment: & is hidden. On the other hand, \begin{matrix}abc\string &\end{matrix} will not show &, but a $ (the first character in the end-of-cell material). For more info on that, see Philippe Goutet's description of \alignsafe@testopt.

Anyways, one solution is to put all of the xstring commands within a brace group, and expand \fooo after it: something like

\begin{matrix}
  {\StrSubstitute{1:2-3:4}{:}{\string&}[\fooo]
  \StrSubstitute{\fooo}{-}{\string\\}[\fooo]
  \tokenize{\fooo}{\fooo}
  \expandafter}\fooo
\end{matrix}

(wrapped in a command if you like). The drawback with this is that we are giving essentially {}1&2\\3&4 to typeset for TeX, and the empty brace group may affect spacing in some cases. Instead, I'd use "fake" braces \iffalse{\fi and \iffalse}\fi, which make TeX think that & isn't the end of a cell, but which don't insert anything to be typeset. The extra \relax in the code below is needed because otherwise TeX, expanding fully the material at the start of the cell, expands \iffalse{\fi too early, when it is not yet recording the number of braces.

\documentclass{article}
\usepackage{mathtools}
\usepackage{xstring}
\newcommand{\foo}[1]{%
    \relax\iffalse{\fi            % <= here
    \StrSubstitute{#1}{:}{\string&}[\fooo]%
    \StrSubstitute{\fooo}{-}{\string\\}[\fooo]%
    \tokenize{\fooo}{\fooo}%
    \iffalse}\fi                  % <= and there
    \fooo}
\begin{document}
\[\begin{matrix}
    \foo{1:2-2:3}
\end{matrix}\]
\end{document}

The other answers point to the fact that you don't need the \tokenize step if you replace \string& by & and \string\\ by \\ (or \tabularnewline), and are careful with expansion. Then, it turns out that & is never seen outside braces (only within \def\...{...&...} and such), and this is why TeX does not choke.

EDIT2: LaTeX3 has a function similar to \tokenize (namely, \tl_gset_rescan:Nnn and friends), and they had the same "bug" as \tokenize (the fact that the user-provided argument is not always hidden behind braces. That's fixed in the svn as of December 18 (today).

Answer 2

Try this code in which : and , are replaced with a & :

\documentclass[]{article}
\usepackage{mathtools}
\usepackage{xstring}
\newcommand\foo[1]{%
    \saveexpandmode\expandarg
    \StrSubstitute{\noexpand#1}:&[\fooo]%
    \StrSubstitute\fooo,&[\fooo]%
    \StrSubstitute\fooo-{\noexpand\\}[\fooo]%
    \restoreexpandmode\[\begin{matrix}\fooo\end{matrix}\]}
\begin{document}
\foo{1:2-2:3-x,y}
\end{document}

Answer 3

You can try the following one:

\documentclass[]{article}
\usepackage{mathtools}
\usepackage{xstring}

\newcommand{\foo}[1]{%
\begingroup
\catcode`\&=12
\noexpandarg
    \StrSubstitute{#1}{-}{\tabularnewline}[\fooo]
\expandarg
    \StrSubstitute{\fooo}{:}{&}[\fooo]
\expandafter\endgroup
    \fooo}


\begin{document}


\[\begin{matrix}
    \foo{1:2-2:3-4444:55555}
\end{matrix}\]
\end{document}