Symbolic functions: changing catcodes and \tl_rescan:nn

Question

The goal is to have the user input (for the sake of argument) a linear function in one variable, for example 2*x+3 along with the variable (this isn't crucial) and a value for the variable. I would then like to calculate the value of the function. The following works, but I'm thinking now that it works more by accident than anything else.

1. My original attempt \findvalueB failed and didn't seem to like the underscore, so I tried the method of \findvalueA which worked. Why does one of these work and the other not?

2. When deciding to switch variable names to test, I ran into the obvious problems (e.g. using y breaks \mytoks. So I tried to hardcode a variable name, again failure. The basic idea seems to work as shown at the bottom. Why does this try fail?

Presumably there's a common thread to what I'm not getting and it's likely a fundamental misunderstanding of something that I'd like to correct. I'm more interested in the explanation than a code solution although both would be nice.

\documentclass{article}
\usepackage{xparse}
\ExplSyntaxOn
\newtoks\mytoks

\NewDocumentCommand {\findvalueA} { m m m }
    {
        \group_begin:
        \char_set_catcode_active:N #1
        \tl_rescan:nn{}{\def#1{#2}}
        \tl_rescan:nn{}{\mytoks={#3}}
        \fp_set:Nn \l_tmpa_fp {\the\mytoks}
        \fp_use:N \l_tmpa_fp
        \group_end:
    }

\NewDocumentCommand {\findvalueB} { m m m }
    {
        \group_begin:
        \char_set_catcode_active:N #1
        \tl_rescan:nn{}{\def#1{#2}}
        % underscores=bad
        \tl_rescan:nn{}{\tl_set:Nn \l_tmpa_tl {#3}}
        % or
        % \tl_rescan:nn{}{\fp_eval:n {#3}}
        \fp_use:N \l_tmpa_fp
        \group_end:
    }

    \NewDocumentCommand {\findvalueC} { m m }
    {
        \group_begin:
        \char_set_catcode_active:N x
        \tl_rescan:nn {} {\defx{#1}}
        \tl_rescan:nn {} {\mytoks={#2}}
        \fp_set:Nn \l_tmpa_fp {\the\mytoks}
        \fp_use:N \l_tmpa_fp
        \group_end:
    }

\ExplSyntaxOff
\begin{document}

% works
\findvalueA{x}{5}{2*x+3}

% fails
%\findvalueB{x}{5}{2*x+3}

% fails
%\findvalueC{5}{x+2}

The method of findvalueC works down here but not above: if $x=5$, then $x+2=$
\ExplSyntaxOn
\group_begin:
\char_set_catcode_active:N x
\defx{5}
\fp_eval:n {x+2}
\group_end:
\ExplSyntaxOff

\end{document}

Answer 1

The failure of method B is pretty easy to explain. Rescanning material reads it with the currently applicable category codes. When you use \findvalueB, you are in the document and so _has category 'math subscript', not 'letter' (you'd also find that : is wrong, a little later). People are often surprised by this as they imagine that the rescan will use the codes set when the command was defined. This is one thing that makes using rescanning quite tricky.

The failure of method C is more difficult to pick up. What happens here is that TeX 'helpfully' inserts a space after \defx when it does the rescan, in the same way it does after any other control sequence when altering tokenization (for example in \write or \detokenize). This happens 'before' the rescan, where \defx will be a single token. The result is that in method C x is defined as being followed by a required space: there's no space, so we get the error. That does not happen in method A as the space here is inserted after \def and before #1 (they are separate tokens before the rescan). Thus this method will work.

Both of these issues show why rescanning is not favoured by a lot of TeX programmers: I tend to avoid it. We have provided reasonable wrappers in LaTeX3, but your problems show that it's still tricky to get right. What you actually want is to replace all of the variable letters with the value. I would therefore not rescan at all, but would use

\NewDocumentCommand { \findvalueD } { m m m }
  {
    \tl_set:Nn \l_tmpa_tl {#3}
    \tl_replace_all:Nnn \l_tmpa_tl {#1} {#2}
    \fp_eval:n { \l_tmpa_tl }
  }

which does not have any of these problems. (I'm assuming that you don't have for example sin in the expression and the letter n as a possible variable: such a case could be handled, but would need more thought!)