Highlight text in code listing while also keeping syntax highlighting

Question

I’m using the listings package for code listings. Now I try to highlight some part of the code using \hl from the soul package.

Here’s an example of a resulting query where the injected part of a SQL injection should be highlighted:

\begin{lstlisting}[escapechar=@,language=SQL]
SELECT name, password FROM users WHERE name='@\hl{' UNION SELECT "10", 1 \#}@';
\end{lstlisting}

But inside the highlighted part the SQL syntax highlighting is not applied.

So I tried to use \lstlinline for the code part inside \hl as well but it didn’t work:

\begin{lstlisting}[escapechar=@,language=SQL]
SELECT name, password FROM users WHERE name='@\hl{\lstinline[language=SQL]$' UNION SELECT "10", 1 #$}@';
\end{lstlisting}

Although this \lstinline part works outside the lstlisting block, it doesn’t work inside it. The error I get for that particular line is:

Extra }, or forgotten \endgroup.

Maybe I’m just missing just some special syntax to get this working. Or is it not possible at all to get proper syntax highlighting for the \hl-highlighted part as well?

Answer 1

It's almost a year later, but as the OP has not yet accepted an answer, I still see my chance :-)

In the following, I use the same basic principle as in my other answer, that is, employ listings moredelim=** option to define delimiters, which styles then apply on top of all other formattings, so that the syntax formatting is kept.

However, instead of being limited to a standard font command for the delimiter style, I now use an lrbox to grab the content of the current group. The resulting box holds the (syntax-formatted) output of listings, which can be used to implement more fancy highlighting styles.

In the example code, the implementation of the \btHL command employs TikZ to typeset the lrbox into a TikZ node. Additional TikZ/PGF options to both, the tikzpicture and the node can be given in the optional argument (such as \btHL[fill=red!20,draw=red]), which provides for pretty flexible highlighters.

(Note: when the optional argument is used inside some listings argument, the whole \btHL[<key>=<value>,...] command has to be put into curly braces in order to not confuse the listings key=value parser.)

The \btHL command can as well be used inside normal text. It does, however, not work across line breaks, which is a possible disadvantage compared to the approach that culminated from Bens answer. Insides of listings, however, this usually is not an issue.

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{listings,xcolor,beramono}
\usepackage{tikz}


\makeatletter
\newenvironment{btHighlight}[1][]
{\begingroup\tikzset{bt@Highlight@par/.style={#1}}\begin{lrbox}{\@tempboxa}}
{\end{lrbox}\bt@HL@box[bt@Highlight@par]{\@tempboxa}\endgroup}

\newcommand\btHL[1][]{%
  \begin{btHighlight}[#1]\bgroup\aftergroup\bt@HL@endenv%
}
\def\bt@HL@endenv{%
  \end{btHighlight}%   
  \egroup
}
\newcommand{\bt@HL@box}[2][]{%
  \tikz[#1]{%
    \pgfpathrectangle{\pgfpoint{1pt}{0pt}}{\pgfpoint{\wd #2}{\ht #2}}%
    \pgfusepath{use as bounding box}%
    \node[anchor=base west, fill=orange!30,outer sep=0pt,inner xsep=1pt, inner ysep=0pt, rounded corners=3pt, minimum height=\ht\strutbox+1pt,#1]{\raisebox{1pt}{\strut}\strut\usebox{#2}};
  }%
}
\makeatother

\lstdefinestyle{SQL}{
    language={SQL},basicstyle=\ttfamily, 
    moredelim=**[is][\btHL]{`}{`},
    moredelim=**[is][{\btHL[fill=green!30,draw=red,dashed,thin]}]{@}{@},
}

\begin{document}

A listing with {\btHL highlighting of all \textbf{important} elements} looks as follows:

\begin{lstlisting}[style=SQL]
SELECT name, password `FROM` users @WHERE@ name=@UNION SELECT@
\end{lstlisting}

Answer 2

Ulrike's suggestion is extremely clever. My version is inspired by what @Ulrike did, but has some improvements:

• The style of the highlight shading is explicitly factored out into \tikzstyle{highlighter} = [...] to clearly show how the visual appearance can be customized. For example, two commented-out lines in my solution show how to make the highlight swipe a bit wavy for a more human-looking result.

• No extra whitespace is added at the start or end of the highlighted region. This required avoiding line breaks in macro definitions, creating the tikz anchor points using \coordinate instead of \node, and creating all tikz diagrams as overlay diagrams.

• The highlighting stroke is added below the text rather than above it. This keeps highlights from affecting the text ink color. Black text with a yellow highlight remains purely black instead of becoming dark yellow. Note that this also means that highlighting does not work inside a listing that has a background colour set since the highlighting stroke will be behind the background.

• Some common code shared by the \bh and \eh macros is factored out into a new \tikzhighlight macro. No visual effect from this, but the vertical centering code is subtle enough that I felt it warranted being factored out.

With those enticements in mind, here's my take on Ulrike's solution:

\documentclass{article}

% required packages
\usepackage{atbegshi,ifthen,listings,tikz}

% change this to customize the appearance of the highlight
\tikzstyle{highlighter} = [
  yellow,
  line width = \baselineskip,
]

% enable these two lines for a more human-looking highlight
%\usetikzlibrary{decorations.pathmorphing}
%\tikzstyle{highlighter} += [decorate, decoration = random steps]

% implementation of the core highlighting logic; do not change!
\newcounter{highlight}[page]
\newcommand{\tikzhighlightanchor}[1]{\ensuremath{\vcenter{\hbox{\tikz[remember picture, overlay]{\coordinate (#1 highlight \arabic{highlight});}}}}}
\newcommand{\bh}[0]{\stepcounter{highlight}\tikzhighlightanchor{begin}}
\newcommand{\eh}[0]{\tikzhighlightanchor{end}}
\AtBeginShipout{\AtBeginShipoutUpperLeft{\ifthenelse{\value{highlight} > 0}{\tikz[remember picture, overlay]{\foreach \stroke in {1,...,\arabic{highlight}} \draw[highlighter] (begin highlight \stroke) -- (end highlight \stroke);}}{}}}

\begin{document}

Works in \bh{}plain text too\eh{} (but not across line breaks).

\begin{lstlisting}[escapechar=@, language=SQL, basicstyle=\sffamily, columns=fullflexible]
SELECT name, password FROM users WHERE name='@\bh@' UNION SELECT "10", 1 #@\eh@';
\end{lstlisting}

\end{document}

The only limitation I am aware of, which also applies to Ulrike's version, is that highlighting does not span line breaks correctly.

Answer 3

I don't think that the combination of \hl + \lstinline can work. Both soul and listings analyze and handle the input one token after the other and do a lot of \catcode magic. They will tread on another toes. But you could try something like this:

 \documentclass{article}
  \usepackage{listings,xcolor,tikz}
  \newcommand\bh{\tikz[remember picture]
                    \node (begin highlight) {};
                 }
  \newcommand\eh{\tikz[remember picture]
                 \node (end highlight) {};
                 \tikz[remember picture, overlay] 
                 \draw[yellow,line width=10pt,opacity=0.3] (begin highlight) -- (end
                  highlight);
                 }


 \begin{document}

\bh abc bce bde bde \eh

\begin{lstlisting}[escapechar=@,language=SQL]
WHERE name=@\bh@UNION SELECT@\eh@
\end{lstlisting}

\end{document}

Answer 4

try it with an explit \textbf:

\documentclass [a4paper]{article}
\usepackage[T1]{fontenc}
\usepackage[scaled=0.83]{beramono}
\usepackage {listings,soul}
\begin{document}

\begin{lstlisting}[language=SQL,escapechar=@,language=SQL,basicstyle=\ttfamily]
SELECT name, password FROM users WHERE name='@\hl{' \textbf{UNION SELECT} "10", 1 \#}@';
\end{lstlisting}

\end{document}

Answer 5

If you are not determined to use \hl (or some simulation of it) and could live with some font-based formatting instead, listings moredelim option would be the most elegant solution. With the moredelim=** syntax you can define delimiters which styles apply on top of all other formattings:

\documentclass{article}
\usepackage{listings,xcolor}

\lstdefinestyle{SQL}{
    language={SQL}, 
    moredelim=**[is][\slshape]{`}{`},
    moredelim=**[is][\color{orange}]{@}{@},
}

\begin{document}

\begin{lstlisting}[style=SQL]
SELECT name, password `FROM` users @WHERE@ name=@`UNION SELECT`@
\end{lstlisting}

\end{document}

Unfortunately, as with all formattings in listings, you can only use macros that affect the text inside the current group. Hence \color and font commands work, but \hl cannot be applied in this way.

Answer 6

As all the solutions so far do not allow a highlight to span several lines, I'd like to present a version that will also work with listing's breaklines option.

The idea is based on the one I've used in this answer, which in turn is based on Ben's improvement of Ulrike's version that uses TikZ' remember picture facilities. In short, the whole highlight is not rendered as a single unit but is broken into a lot of small pieces that are then able to get broken across lines with TeX's standard line breaking algorithm. What is a bit unfortunate about this approach is that it limits the number of highlight styles that can be used, because two adjacent highlight segments must fit together visually to produce another continuous segment. This is best demonstrated by modifying the \draw command in the answer mentioned above to render framed nodes instead of line segments (note especially that, depending on the selected language, not even F1 is kept as a single unit):

To overcome this limitation we must eliminate the intermediate nodes such that each highlight on the same line only consists of a single segment. So the basic idea is to remember the position of the start node and check for each intermediate node if we can ignore it or if it is the end of this line's highlight. We use two hooks in \lst@discretionary, one before and one after a potential break. If there's no break, both nodes stay on the same line and we can ignore them. Otherwise the first node will be the temporary end node and the second will start a new highlight on the following line. When the user-defined end node occurs, we know we are finished and must close the last opened highlight.

Implementing this approach is a bit more tricky, because the exact positions of all nodes are not known until the shipout for the current page occurs. That means we need a two-stage process. On the first compilation run all potential nodes are written to the .aux file (through \hl@mark). I dropped the remember picture approach here entirely and used the raw \pdfsavepos because extra information about the node type and page number has to be stored along with the coordinates. This also seems to make the whole processing more performant when a lot of highlights are used in the document. Additionally, the coordinates of the origin are stored for each page and used later for correct positioning of the nodes.

The main logic is in the \hl@processmark macro, which is called for every potential node. It decides which nodes are eventually used for output and which are discarded. If a complete highlight segment is found, the corresponding styling command call is (through \hl@draw) added together with the page number to the \hl@shipout token register. On the next shipout \hl@doshipout then loops through all the saved segments' code and renders everything that should go on the current page.

To allow some type of customization of the output, the \hl@draw command takes a parameter used for styling material. If the parameter is empty, the default styling command \hldefaultstyle is used inside a \tikz call. Otherwise the parameter is used as the command. In both cases four parameters are added to the call: the x- and y-coordinates of the first and last segment's node, respectively. Because the styling parameters have to be passed all the way down to the final shipout, they are stored in the .aux file, too.

For the user interface the following commands are defined:

• \hlstyle can be used for listings delimiter styles. It takes one optional parameter in case a custom styling command should be used.
• \hl is similar to \hlstyle but for uses in inline text. Note that this command doesn't work correctly with line breaks, so it should be used with care.
• \hldefaultstyle is the default styling command used if no custom styling commands are provided.
• \hlunderlinestyle, \hlmarkerstyle, \hlboxstyle are three example styling commands for underlined, marked and framed box output, respectively. Each command takes an optional parameter for the highlighting color.

Now here's the final example document:

\documentclass{article}

\usepackage{listings}
\usepackage{xcolor}
\usepackage{atbegshi}
\usepackage{ifthen}

\usepackage{tikz}
\usetikzlibrary{calc}
\usetikzlibrary{fit}

\makeatletter

\newcommand\if@empty[1]{%
    \if\relax\detokenize{#1}\relax
        \expandafter\@firstoftwo
    \else
        \expandafter\@secondoftwo
    \fi
}

\newcommand\ifthen[2]{\ifthenelse{#1}{#2}{}}
\newcommand\ifelse[2]{\ifthenelse{#1}{}{#2}}

% Is highlighting currently active in the listings environment?
\newif\ifhl@active

% Material executed at each page shipout
\newtoks\hl@shipout

% Extra styling parameters for the currently active highlight
\newtoks\hl@parameters

% Inline highlight style. This does not break across line ends
\newcommand\hl[2][]{\begingroup\hlstyle@start{#1}#2\endgroup}

% The highlight style for listings environments
\newcommand\hlstyle[1][]{\hlstyle@start{#1}}
\newcommand\hlstyle@start[1]{%
    \global\hl@activetrue
    \aftergroup\hlstyle@end
    \write\@auxout{\noexpand\hl@setparameters{#1}}%
    \hl@mark{S}%
}
\newcommand\hlstyle@end{%
    \hl@mark{E}%
    \global\hl@activefalse
}

% Sets the current highlight parameters
\newcommand\hl@setparameters[1]{%
    \global\hl@parameters={#1}%
}

% Writes a new node marker with the current position and page number to the .aux file
\newcommand\hl@mark[1]{%
    \ensuremath{\vcenter{\hbox{\pdfsavepos}}}%
    \write\@auxout{\noexpand\hl@processmark{#1}{\the\pdflastxpos}{\the\pdflastypos}{\arabic{page}}}%
}

% Writes the current page origin to the .aux file, and issues the current \hl@shipout material
% to be processed
\AtBeginShipout{\AtBeginShipoutUpperLeft{%
    \pdfsavepos
    \write\@auxout{\noexpand\hl@processmark{Z}{\the\pdflastxpos}{\the\pdflastypos}{\arabic{page}}}%
    \hl@doshipout
}}

% Maintains the current first and last node positions, checks whether a node marker should be used
% for the current highlighting segment, and issues a \hl@draw call if necessary.
% #1 is either 'S' (start node), 'M' (intermediate node), 'E' (end node), or 'Z' (page origin)
% #2/#3 are the x/y-coordinates, #4 is the page number
\newcommand\hl@processmark[4]{%
    \if#1S%
        \xdef\hl@firstx{#2}%
        \xdef\hl@firsty{#3}%
        \xdef\hl@firstp{#4}%
    \else\if#1M%
        \ifthenelse{\hl@firsty=#3}{%
            % Still on same line
            \xdef\hl@lastx{#2}%
            \xdef\hl@lasty{#3}%
        }{%
            \ifelse{\hl@firstx=\hl@lastx \and \hl@firsty=\hl@lasty}{%
                % Last highlight part is not empty
                \expandafter\hl@draw\expandafter{\the\hl@parameters}%
            }%
            \xdef\hl@firstx{#2}%
            \xdef\hl@firsty{#3}%
            \xdef\hl@firstp{#4}%
        }%
    \else\if#1E%
        \ifthen{\hl@firsty=#3}{%
            % Still on same line
            \xdef\hl@lastx{#2}%
            \xdef\hl@lasty{#3}%
            \ifelse{\hl@firstx=#2}{%
                % Last highlight part is not empty
                \expandafter\hl@draw\expandafter{\the\hl@parameters}%
            }%
        }%
    \else\if#1Z%
        \xdef\hl@zerox{#2}%
        \xdef\hl@zeroy{#3}%
    \fi\fi\fi\fi
}

% Add a new highlight segment to \hl@shipout to be processed on every shipout.
% #1 is the styling command used to draw the segment within a properly positioned \tikz call.
%    If #1 is empty, \hldefaultstyle is used as a default. Every styling command can take
%    arbitrary parameters, only the last four are mandatory (x_first, y_first, x_last, y_last)
\newcommand\hl@draw[1]{%
    \edef\@temp{{\noexpand\tikz [ overlay, shift = {(-\hl@zerox sp, -\hl@zeroy sp)} ]
        {\if@empty{#1}{\noexpand\hldefaultstyle}{\unexpanded{#1}}%
            {\hl@firstx sp}{\hl@firsty sp}{\hl@lastx sp}{\hl@lasty sp}}}%
        {\hl@firstp}%
    }%
    \expandafter\global\expandafter\addto@hook\expandafter\hl@shipout\expandafter{\@temp}%
}

% Processes the current \hl@shipout material. The material is a sequence of
% {TikZ commands}{page number} pairs
\def\hl@doshipout{%
    \expandafter\hl@doshipout@\the\hl@shipout{}{}\@end
}
\def\hl@doshipout@#1#2#3\@end{%
    \if@empty{#2}{}{%
        \ifthen{\arabic{page}=#2}{#1}%
        \hl@doshipout@#3\@end
    }%
}

% Hook into listings' discretionary breaks
\let\orig@lst@discretionary=\lst@discretionary
\gdef\lst@discretionary{%
    \ifhl@active
        \hl@mark{M}%
    \fi
    \orig@lst@discretionary
    \ifhl@active
        \hl@mark{M}%
    \fi
}


% Styling command for underlined text
\newcommand\hlunderlinestyle[5][black]{%
    \draw [ yshift = -0.4\baselineskip, #1 ] (#2, #3) -- (#4, #5);
}

% Styling command for a simple marker highlights
\newcommand\hlmarkerstyle[5][yellow]{%
    \draw [ line width = \baselineskip, #1 ] (#2, #3) -- (#4, #5);
}

% Styling command for a framed box
\newcommand\hlboxstyle[5][red]{%
    \draw let \p1 = (#2, #3), \p2 = (#4, #5)
          in node [ draw = #1,
                    fill = #1!10!white,
                    minimum height = \baselineskip,
                    fit = (\p1)(\p2),
                    anchor = west,
                    align = none,  % prevent underfull node boxes
                    outer sep = 0pt, inner sep = 0pt ]
             at (\p1) {};
}

% Default styling command
\let\hldefaultstyle=\hlmarkerstyle

\makeatother


\newcommand\delimstyleA{\hlstyle[{\hlboxstyle[green]}]}
\newcommand\delimstyleB{\hlstyle}
\newcommand\delimstyleC{\hlstyle[{\hlunderlinestyle[red, line width = 0.8pt]}]}
\lstset{
    language=SQL,
    columns=fullflexible,
    basicstyle=\sffamily,
    moredelim=**[is][\delimstyleA]{@}{@},
    moredelim=**[is][\delimstyleB]{`}{`},
    moredelim=**[is][\delimstyleC]{_}{_},
    keepspaces,
    breaklines
}

\begin{document}
\vspace*{15.5cm}
Example listing for \hl[{\hlboxstyle[blue]}]{SQL}'s \lstinline|_SELECT_| statement:
\begin{lstlisting}
SELECT name, `password`, _email_ FROM users
    WHERE name=@'UNION SELECT "10", 1'@;
SELECT name, `password`, _email_ FROM users WHERE name=_'UNION SELECT "10", 1'_;
SELECT name, `password`, _email_ FROM users WHERE name=@'UNION SELECT x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, x14, x15, x16, x17, x18, x19, x20, x21, x22, x23, x24, x25, x26, x27, x28, x29, x30, x31, x32, x33, x34, x35, x36, x37, x38, x39, x40'@;
\end{lstlisting}
\end{document}

---

Answer 7

If PDF annotations are okay, you can give the pdfcomment package a try. Then you can also add a comment.

\documentclass[a4paper]{article}
\usepackage[T1]{fontenc}
\usepackage[scaled=0.83]{beramono}
\usepackage{listings,xcolor,pdfcomment}
\begin{document}
\begin{lstlisting}[language=SQL,escapechar=@,language=SQL,basicstyle=\ttfamily]
SELECT name, password FROM users WHERE name='@\pdfmarkupcomment[markup=Underline,color=black,author=Gumbo]{' \textbf{UNION SELECT} "10", 1 \#}{This is the injected part of a SQL injection}@';
\end{lstlisting}
\end{document}