Symbolic functions: changing catcodes and \tl_rescan:nn

The goal is to have the user input (for the sake of argument) a linear function in one variable, for example 2*x+3 along with the variable (this isn't crucial) and a value for the variable. I would then like to calculate the value of the function. The following works, but I'm thinking now that it works more by accident than anything else.

1. My original attempt \findvalueB failed and didn't seem to like the underscore, so I tried the method of \findvalueA which worked. Why does one of these work and the other not?

2. When deciding to switch variable names to test, I ran into the obvious problems (e.g. using y breaks \mytoks. So I tried to hardcode a variable name, again failure. The basic idea seems to work as shown at the bottom. Why does this try fail?

Presumably there's a common thread to what I'm not getting and it's likely a fundamental misunderstanding of something that I'd like to correct. I'm more interested in the explanation than a code solution although both would be nice.

\documentclass{article}
\usepackage{xparse}
\ExplSyntaxOn
\newtoks\mytoks

\NewDocumentCommand {\findvalueA} { m m m }
{
\group_begin:
\char_set_catcode_active:N #1
\tl_rescan:nn{}{\def#1{#2}}
\tl_rescan:nn{}{\mytoks={#3}}
\fp_set:Nn \l_tmpa_fp {\the\mytoks}
\fp_use:N \l_tmpa_fp
\group_end:
}

\NewDocumentCommand {\findvalueB} { m m m }
{
\group_begin:
\char_set_catcode_active:N #1
\tl_rescan:nn{}{\def#1{#2}}
\tl_rescan:nn{}{\tl_set:Nn \l_tmpa_tl {#3}}
% or
% \tl_rescan:nn{}{\fp_eval:n {#3}}
\fp_use:N \l_tmpa_fp
\group_end:
}

\NewDocumentCommand {\findvalueC} { m m }
{
\group_begin:
\char_set_catcode_active:N x
\tl_rescan:nn {} {\defx{#1}}
\tl_rescan:nn {} {\mytoks={#2}}
\fp_set:Nn \l_tmpa_fp {\the\mytoks}
\fp_use:N \l_tmpa_fp
\group_end:
}

\ExplSyntaxOff
\begin{document}

% works
\findvalueA{x}{5}{2*x+3}

% fails
%\findvalueB{x}{5}{2*x+3}

% fails
%\findvalueC{5}{x+2}

The method of findvalueC works down here but not above: if $x=5$, then $x+2=$
\ExplSyntaxOn
\group_begin:
\char_set_catcode_active:N x
\defx{5}
\fp_eval:n {x+2}
\group_end:
\ExplSyntaxOff

\end{document}

-

The failure of method B is pretty easy to explain. Rescanning material reads it with the currently applicable category codes. When you use \findvalueB, you are in the document and so _has category 'math subscript', not 'letter' (you'd also find that : is wrong, a little later). People are often surprised by this as they imagine that the rescan will use the codes set when the command was defined. This is one thing that makes using rescanning quite tricky.

The failure of method C is more difficult to pick up. What happens here is that TeX 'helpfully' inserts a space after \defx when it does the rescan, in the same way it does after any other control sequence when altering tokenization (for example in \write or \detokenize). This happens 'before' the rescan, where \defx will be a single token. The result is that in method C x is defined as being followed by a required space: there's no space, so we get the error. That does not happen in method A as the space here is inserted after \def and before #1 (they are separate tokens before the rescan). Thus this method will work.

Both of these issues show why rescanning is not favoured by a lot of TeX programmers: I tend to avoid it. We have provided reasonable wrappers in LaTeX3, but your problems show that it's still tricky to get right. What you actually want is to replace all of the variable letters with the value. I would therefore not rescan at all, but would use

\NewDocumentCommand { \findvalueD } { m m m }
{
\tl_set:Nn \l_tmpa_tl {#3}
\tl_replace_all:Nnn \l_tmpa_tl {#1} {#2}
\fp_eval:n { \l_tmpa_tl }
}


which does not have any of these problems. (I'm assuming that you don't have for example sin in the expression and the letter n as a possible variable: such a case could be handled, but would need more thought!)

-
Aha! Sneaky TeX, it's so hard to diagnose things sometimes! I suppose the problem with B should have suggested itself, but I never would have known what the problem with C was. It seems like that '\tl_replace_all:Nnn is in one of my blind spots, it's such a useful command but I always overlook it. Twice now it's been an easy answer to one of my questions. Thanks for the answer. –  Scott H. Aug 27 '12 at 16:54
@ScottH. \fp_trap:nn is not a command we define: perhaps you mean \fp_eval:n. This was only introduced recently (we track version by SVN revision, so I'd need to know what revision of expl3 you have to say if you should have this or not). –  Joseph Wright Aug 27 '12 at 20:35
@ScottH. Ah, one of Bruno's experimental things. They may well be buggy at present. I would suggest you raise this on the LaTeX-L mailing list or as a separate question here, and I'll prod him to take a look. –  Joseph Wright Aug 27 '12 at 20:43
@ScottH. Bruno has now added some experimental code which allows 'automagic' conversion of symbolic variables, e.g. \fp_var_set:nn { x } { 5 } followed by \fp_eval:n { x + 2 }`. Not yet released, but you can grab it from the GitHub site (github.com/latex3/svn-mirror) if you're interested. –  Joseph Wright Aug 30 '12 at 19:48
Ooh! That sounds great, thanks for the heads up :) –  Scott H. Aug 30 '12 at 20:35