fancyvrb error with math escapes and prime (')

Question

I've recently noticed a few cases where fancyvrb math escapes fail. This works fine:

\documentclass{article}
\usepackage{fancyvrb}
\begin{document}
\begin{Verbatim}[commandchars=\\\{\},codes={\catcode`\$=3\catcode`\^=7\catcode`\_=8}]
$f^\prime$
\end{Verbatim}
\end{document}

But substituting this

\begin{Verbatim}[commandchars=\\\{\},codes={\catcode`\$=3\catcode`\^=7\catcode`\_=8}]
$f'$
\end{Verbatim}

fails with the message

! TeX capacity exceeded, sorry [main memory size=5000000].
'->\leavevmode \kern \z@ 
                         \char `\'

The same sort of error can be produced by trying to use icomma with fancyvrb.

It isn't hard to work around this, but it would be nice if escaped math didn't have to be checked for such conflicts. I believe this has something to do with the interaction of fancyvrb and mathcodes, but I haven't made progress in seeing how that differs from the catcode case.

Edit

egreg's answer explains things. For anyone working with this in the future, here are a few additional notes.

• See this on the use of \expandafter in \fi*...\fi.
• See this for more on \do@noligs.

Finally, since fancyvrb is often used with upquote, here is a version of egreg's code that patches the Verbatim environment to give primes in math mode and upquote single quotes otherwise. Given the way upquote works, both ' and ` need to be redefined. This requires etoolbox.

\makeatletter
\AtEndPreamble{%
  \@ifpackageloaded{upquote}%
   {\AtBeginEnvironment{Verbatim}{%
      \apptocmd{\@noligs}%
       {\begingroup\lccode`\~=`\'
        \lowercase{\endgroup\def~}{%
          \ifmmode
            \expandafter\active@math@prime
          \else
            \leavevmode\kern\z@\textquotesingle
          \fi}%
        \ifx\encodingdefault\upquote@OTone
        \ifx\ttdefault\upquote@cmtt
        \begingroup\lccode`\~=`\'
        \lowercase{\endgroup\def~}{%
          \ifmmode
            \expandafter\active@math@prime
          \else
            \leavevmode\kern\z@\char13
          \fi}%
        \fi\fi
        \begingroup\lccode`\~=`\`
        \lowercase{\endgroup\def~}{%
          \ifmmode
            \expandafter`
          \else
            \leavevmode\kern\z@\textasciigrave
          \fi}%
        \ifx\encodingdefault\upquote@OTone
        \ifx\ttdefault\upquote@cmtt
        \begingroup\lccode`\~=`\`
        \lowercase{\endgroup\def~}{%
          \ifmmode
            \expandafter`
          \else
            \leavevmode\kern\z@\char18
          \fi}%
        \fi\fi
        }{}{}}}%
   {\AtBeginEnvironment{Verbatim}{%
      \apptocmd{\@noligs}%
       {\begingroup\lccode`\~=`\'
        \lowercase{\endgroup\def~}{%
          \ifmmode
            \expandafter\active@math@prime
          \else
            \leavevmode\kern\z@\char`\'
          \fi}}{}{}}}%
}
\makeatother

Answer 1

In verbatim modes, the active ' is redefined to do

\leavevmode\kern\z@\char`'

and this will be the definition picked up when $f'$ is found in such a verbatim mode, even if escaped.

You have to make the definition of the active ' to be the usual one in math mode.

\documentclass{article}
\usepackage{fancyvrb,etoolbox}

\makeatletter
\apptocmd\do@noligs
  {\begingroup\lccode`\~=`\'
   \lowercase{\endgroup\def~}{%
     \ifmmode
       \expandafter\active@math@prime
     \else
       \leavevmode\kern\z@\char`\'
     \fi}}
  {}{}
\makeatother

\begin{document}
\begin{Verbatim}[commandchars=\\\{\},codes={\catcode`\$=3\catcode`\^=7\catcode`\_=8 }]
$f^\prime$ and $f'$
\end{Verbatim}
\end{document}