# Replace chars with xstring. Catcode problem

I try to do something like this (minimal example):

\documentclass[]{article}
\usepackage{mathtools}
\usepackage{xstring}

\newcommand{\foo}[1]{
\StrSubstitute{#1}{:}{\string&}[\fooo]
\StrSubstitute{\fooo}{-}{\string\\}[\fooo]
\tokenize{\fooo}{\fooo}
\fooo}

\begin{document}

$\begin{matrix} \foo{1:2-2:3} \end{matrix}$
\end{document}


But I it isn't working, for some reason this is working:

\documentclass[]{article}
\usepackage{mathtools}
\usepackage{xstring}

\begin{document}

\StrSubstitute{1:2-2:3}{:}{\string&}[\fooo]
\StrSubstitute{\fooo}{-}{\string\\}[\fooo]
\tokenize{\fooo}{\fooo}

$\begin{matrix} \fooo \end{matrix}$
\end{document}


Any Idea on this? I think this is related to catcodes...

Maybe someone also has an explanation for this behavior?

-

The reason your code doesn't work is that the matrix environment (or rather, the underlying \halign TeX primitive) sees & while xstring is manipulating it (more precisely, while performing the \tokenize step). TeX thinks that this is the end of the cell, inserts the end-of-cell material, including the internal primitive \endtemplate, which makes the xstring macros choke. If you want to see this in action, add \tracingall before \tokenize, and get a confusing output.

So... how do we hide & from TeX's inquisitive eyes? It turns out that if TeX sees & (or rather, a token with catcode 4, alignment tab) when it thinks that it is within braces, the & token is left as it is, and not converted to the end of a cell. Thus for instance \def\foo{&} will not make TeX choke when appearing in an alignment: & is hidden. On the other hand, \begin{matrix}abc\string &\end{matrix} will not show &, but a \$ (the first character in the end-of-cell material). For more info on that, see Philippe Goutet's description of \alignsafe@testopt.

Anyways, one solution is to put all of the xstring commands within a brace group, and expand \fooo after it: something like

\begin{matrix}
{\StrSubstitute{1:2-3:4}{:}{\string&}[\fooo]
\StrSubstitute{\fooo}{-}{\string\\}[\fooo]
\tokenize{\fooo}{\fooo}
\expandafter}\fooo
\end{matrix}


(wrapped in a command if you like). The drawback with this is that we are giving essentially {}1&2\\3&4 to typeset for TeX, and the empty brace group may affect spacing in some cases. Instead, I'd use "fake" braces \iffalse{\fi and \iffalse}\fi, which make TeX think that & isn't the end of a cell, but which don't insert anything to be typeset. The extra \relax in the code below is needed because otherwise TeX, expanding fully the material at the start of the cell, expands \iffalse{\fi too early, when it is not yet recording the number of braces.

\documentclass{article}
\usepackage{mathtools}
\usepackage{xstring}
\newcommand{\foo}[1]{%
\relax\iffalse{\fi            % <= here
\StrSubstitute{#1}{:}{\string&}[\fooo]%
\StrSubstitute{\fooo}{-}{\string\\}[\fooo]%
\tokenize{\fooo}{\fooo}%
\iffalse}\fi                  % <= and there
\fooo}
\begin{document}
$\begin{matrix} \foo{1:2-2:3} \end{matrix}$
\end{document}


The other answers point to the fact that you don't need the \tokenize step if you replace \string& by & and \string\\ by \\ (or \tabularnewline), and are careful with expansion. Then, it turns out that & is never seen outside braces (only within \def\...{...&...} and such), and this is why TeX does not choke.

EDIT2: LaTeX3 has a function similar to \tokenize (namely, \tl_gset_rescan:Nnn and friends), and they had the same "bug" as \tokenize (the fact that the user-provided argument is not always hidden behind braces. That's fixed in the svn as of December 18 (today).

-
 Thanks for your great explication! I got an Idea of the Problem and your proposed solution. It is working, as far as I can tell. Will do some more tests tomorrow. I'm just wondering if it would be possible to dedect a normally typed new line and replace it with \\ – someonr Dec 19 '11 at 0:09 It is impossible to detect new lines reliably: they are converted to spaces very early on. There was a question some time ago about how to define a macro delimited by line breaks, but I can't find it :(. – Bruno Le Floch Dec 19 '11 at 2:19 thx, I thought that would be possible, because I was thinking that environments like verbatim also have to do something like that. (They don't need an explicit line break with \) – someonr Dec 19 '11 at 12:04 @someonr: it is possible to detect new lines unreliably. If you really want that, you can ask a separate question: surely someone will know where to point you to. The details are a little bit messy. – Bruno Le Floch Dec 19 '11 at 12:34

Try this code in which : and , are replaced with a & :

\documentclass[]{article}
\usepackage{mathtools}
\usepackage{xstring}
\newcommand\foo[1]{%
\saveexpandmode\expandarg
\StrSubstitute{\noexpand#1}:&[\fooo]%
\StrSubstitute\fooo,&[\fooo]%
\StrSubstitute\fooo-{\noexpand\\}[\fooo]%
\restoreexpandmode$\begin{matrix}\fooo\end{matrix}$}
\begin{document}
\foo{1:2-2:3-x,y}
\end{document}

-
you should use \unexpanded{#1} if you want to prevent the expansion of the user's argument. Otherwise, only the first token is notexpanded. – Bruno Le Floch Dec 18 '11 at 22:13
No. As the manual says, only the first token is expanded when \expandarg mode. Therefore, a simple \noexpand prevents this expansion. – unbonpetit Dec 18 '11 at 22:17
Sorry, I haven't read the xstring manual, I thought it woule be an edef. --- IIRC, you're the xstring maintainer. You should add \relax\iffalse{\fi\ifnum\}=0\fi at the start of the definition for \tokenize and \ifnum\{=0\fi\iffalse}\fi at the end (or similar, see Philippe Goutet's answer I link to in my answer). – Bruno Le Floch Dec 18 '11 at 22:23
Arguments are expanded with \edef when under \fullexpandarg mode. For \tokenize, you see that it is just a simple way to use \scantokens. Though I read the link you suggested, I do not see why I should add something to the definition of \tokenize, both at its begining and at its end. – unbonpetit Dec 18 '11 at 22:42
In the current state, \tokenize{\foo}{&} will fail in an alignment context because & is not "hidden behind braces" (the master counter is zero when & is read). The solution which changes the least amount of code is to add \iffalse{\fi and \iffalse}\fi to \tokenize, so that unsuspecting users doing \tokenize\foo{&} will not trigger TeX's wrath: the master counter is then 1 when & is read. See my email for more details, and other solutions. – Bruno Le Floch Dec 19 '11 at 0:04

You can try the following one:

\documentclass[]{article}
\usepackage{mathtools}
\usepackage{xstring}

\newcommand{\foo}[1]{%
\begingroup
\catcode\&=12
\noexpandarg
\StrSubstitute{#1}{-}{\tabularnewline}[\fooo]
\expandarg
\StrSubstitute{\fooo}{:}{&}[\fooo]
\expandafter\endgroup
\fooo}

\begin{document}

$\begin{matrix} \foo{1:2-2:3-4444:55555} \end{matrix}$
\end{document}
`
-
 Your Code works good. What would I do if I need more than one Symbol replaced by &? – someonr Dec 18 '11 at 14:08 @someonr: I can't understand the question. Can you provide a small example. – Marco Daniel Dec 18 '11 at 19:39 never mind, thx for your answer. I think I just understood the problem, thx to the answer from Bruno Le Floch. – someonr Dec 19 '11 at 0:07