# Expand macro inside lstlisting and apply syntax highlighting

I'm wondering if there's a way to expand a macro inside a lstlisting environment which automatically applies syntax highlighting to the expanded text.

For example, the listing in

\documentclass{article}
\usepackage{listings}
\usepackage{xcolor}

\lstset{
basicstyle=\ttfamily,
keywordstyle=\color{blue},
keywords={function},
escapeinside={|}{|}
}
\def\fbar{function bar ();}

\begin{document}
\begin{lstlisting}[]
function foo ();
|\fbar|
\end{lstlisting}
\end{document}


should produce this output:

There is now! I also hate listings now :)

Well, sort of. Read on and you'll see.

First, the escapechar=<char> option is, for our disappointment, just a shorthand of escapeinside=<char><char>. Not terribly useful, if you ask me. The docs and the sources don't point to anything other than that.

I implemented a cschar=<char> option which will use the <char> as a escape character (in TeX's, not listings' meaning). I recycled some code from listings' escapeinside and made it start a \lst@scan@csname thing. This scanner grabs as much characters between a-z and A-Z (no \m@cr@s allowed) as possible and builds a control sequence out of them, or it grabs one non-letter characters and build a control character. The built control sequence/character is then expanded (once) and retokenized using expl3's \tl_rescan:nn (I didn't have the patience to copy its definition) and reinserted in the token stream for listings to do its thing. If the expanded control sequence happens to have another control sequence inside, it is later expanded as well, working recursively.

Beware that this is an emulation of how TeX builds a control sequence, so it has some bugs features that aren't worth a check. For instance, \fbar hello will print function bar(); hello, with a space between, while TeX would expand to function bar();hello. This makes it impossible for you to get, for instance function bar();hello because \fbarhello would be interpreted as one (undefined) control sequence. You'd need to define a control sequence \def\hello{hello} and use \fbar\hello.

Here's the code:

\documentclass{article}
\usepackage{listings}
\usepackage{xcolor}
\usepackage{expl3}

\makeatletter
\gdef\lst@CSEscape#1{%
\lst@CArgX #1\relax\lst@CDefX
{}%
{\lst@ifdropinput\else
\lst@TrackNewLines\lst@OutputLostSpace \lst@XPrintToken
\lst@InterruptModes
\lst@EnterMode{\lst@TeXmode}{\lst@modetrue}%
\let\do\@makeother\dospecials
\let\lst@grabbed@csname\@empty
\expandafter\lst@scan@csname%
\fi}%
{}}
\lst@Key{cschar}{}
{\ifx\@empty#1\@empty
\let\lst@CSChar\relax
\else
\def\lst@CSChar{\lst@CSEscape{#1}}%
\fi}
\let\lst@explicit@space=\ %
\def\lst@afterfi#1#2\fi{\fi#1}
\def\lst@scan@csname#1{%
\if\relax\detokenize{#1}\relax
\lst@afterfi{\lst@dispatch@csname{#1}}%
\else
\lst@afterfi{%
\uppercase{\def\lst@tmpa{#1}}%
\expandafter\lst@scan@csname@\expandafter
{\number\expandafter\lst@tmpa}{#1}}%
\fi}
\def\lst@scan@csname@#1#2{%
\ifnum %      vv If you want to \maketatletter :)
% \ifnum0#1>63 \expandafter0\else\expandafter1\fi
\ifnum0#1>64 \expandafter0\else\expandafter1\fi
\ifnum0#1<91 \expandafter0\else\expandafter1\fi
=0
\edef\lst@grabbed@csname{\lst@grabbed@csname\string#2}%
\expandafter\lst@scan@csname
\else
\lst@afterfi{\lst@dispatch@csname{#2}}%
\fi}
\def\lst@dispatch@csname#1{%
\ifx\lst@grabbed@csname\@empty
\if\detokenize{#1}\lst@char@space
\lst@afterfi{\lst@afterfi{\lst@return@from@csname@{}{#1}}}%
\else
\lst@afterfi{\lst@afterfi{\lst@dispatch@csname@{\string#1}}}%
\fi
\else
\lst@afterfi{\expandafter\lst@dispatch@csname@\expandafter{\lst@grabbed@csname}{#1}}%
\fi}
\def\lst@dispatch@csname@#1{%
\@ifundefined{#1}%
{\PackageError{Listings}{! Undefined control sequence \@backslashchar#1}}%
{\expandafter\lst@return@from@csname\csname#1\endcsname}}
\def\lst@return@from@csname#1{%
\ifx#1\lst@explicit@space
\lst@afterfi{\lst@return@from@csname@{}{}}%
\else
\lst@afterfi{\expandafter\lst@return@from@csname@\expandafter{#1}}%
\fi}
\def\lst@return@from@csname@#1#2{%
\lst@escapeend \lst@LeaveAllModes\lst@ReenterModes
\lst@newlines\z@ \lst@whitespacefalse
\lst@csescape@output{#1}{#2}}
\def\if@single@item#1{%
\if\relax\detokenize\expandafter{\@gobble#1}\relax
\expandafter\@firstoftwo
\else
\expandafter\@secondofone
\fi}
\def\lst@active@space{\lst@ProcessSpace}
\begingroup
\catcode\|=0
\catcode$=1 \catcode$=2
\let\do\@makeother\dospecials
[|global|let|lst@char@space= ]
|endgroup
\def\lst@csescape@output#1#2{%
\if@single@item{#2}%
{%
\ifnum
\ifx\lst@active@space#2\expandafter1\else\expandafter0\fi
\ifx\lst@char@space#2\expandafter1\else\expandafter0\fi
>0
\def\lst@aftertokens{\safescantokens{#1}\lst@ProcessSpace}%
\else
\def\lst@aftertokens{\safescantokens{#1#2}}%
\fi
}{%
\def\lst@aftertokens{\safescantokens{#1#2}}%
}%
\lst@aftertokens
}
\makeatother
\ExplSyntaxOn
\cs_new_protected:Npn \safescantokens #1 { \tl_rescan:nn {} {#1}}
\ExplSyntaxOff

\begin{document}
\lstset{
basicstyle=\ttfamily,
keywordstyle=\color{blue},
keywords={function},
cschar=\\,
}

\def\fbar{function bar ();}
\def\fuba{cat fuba ();}
\def\1{function first ();}

\begin{lstlisting}[]
function foo ();@12\ 3
\fbar\1{}@12\ 3
\fbar\fuba@12\ 3
\end{lstlisting}

\begin{lstlisting}[]
function foo (); @12\ 3
\fbar{} @12\ 3
\fbar @12\ 3
\fbar\ @12\ 3
\end{lstlisting}

\lstset{cschar=|}

\begin{lstlisting}[]
function foo ();@12\ 3
|fbar{}@12\ 3
|fbar@12\ 3
\end{lstlisting}

\begin{lstlisting}[]
function foo (); @12\ 3
|fbar{} @12\ 3
|fbar @12\ 3
|fbar\ @12\ 3
\end{lstlisting}

\end{document}


and the output:

I'd use this with care. It's the kind of code that doesn't inspire confidence.

P.S.: If you want to make the code know more characters as “letter” you have to change the conditional:

  \ifnum %      vv If you want to \maketatletter :)
% \ifnum0#1>63 \expandafter0\else\expandafter1\fi
\ifnum0#1>64 \expandafter0\else\expandafter1\fi
\ifnum0#1<91 \expandafter0\else\expandafter1\fi
=0


for instance, changing the first 64 to 63 will make @ act as a letter.

• Great work! Perhaps this could even go into the official listings package as a new experimental feature? – siracusa May 8 at 22:17
• @siracusa Thanks :) Modesty aside, I think it's a nice feature, although the code certainly has issues. I may try to contact Jobst if you don't find any obvious problems in this code while using it :) – Phelype Oleinik May 8 at 23:29
\documentclass{article}
\usepackage{listings}
\usepackage{xcolor}

\lstset{
basicstyle=\ttfamily,
keywordstyle=\color{blue},
keywords={function},
escapeinside={|}{|}
}
\def\fbar{function bar ();}

\begin{document}
\begin{lstlisting}[]
function foo ();
|\bgroup\endlinechar=-1 \scantokens\expandafter{\expandafter\egroup\expandafter|\fbar}
function baz ();
\end{lstlisting}
\end{document}


\documentclass{article}
\usepackage{listings}
\usepackage{xcolor}

\lstset{
basicstyle=\ttfamily,
keywordstyle=\color{blue},
keywords={function},
escapeinside={|}{|}
}
\begingroup
\catcode\^^M=12 %
\csname@firstofone\endcsname{%
\endgroup%
\def\fbar{%
function bar ();
function baz ();%
}%
}%
%

\begin{document}
\begin{lstlisting}[]
function foo ();
|\bgroup\endlinechar=-1 \scantokens\expandafter{\expandafter\egroup\expandafter|\fbar} Something in the same line as baz;
function bat ();
\end{lstlisting}
\end{document}
`