Automatic text highlighting based on a dictionary

Question

I would like to automatically highlight parts of a running text, based on whether the word is in a predefined dictionary, or not. Something along the lines of...

\documentclass{article}
\mydictionary{apple,banana}%A comma-separated list of unicode characters. For simplicity the dictionary contains words, and not compound expressions, such as "yellow lemon". In other words, no spaces are allowed.
\def\autohighlightStyle1{ textcolor=red }{ dictionary=\mydictionary }%I need the infrastructure to define my own formatting. For simplicity, in this example it is enough to change the textcolor. I will add my own formatting options later by myself.
\begin{document}
\autohighlightStyle1{Monkeys like banana, but they don't like apple.}
\end{document}

And the result would be a text, where the words from dictionary \mydictionary (banana and apple) are formatted according to some instructions, here for simplicity they are colored to red.

Any chance that something like this is doable? Essentially I will need to tokenize the text by the spaces, and then compare word by word it with the contents of the dictionary, and then apply the style, and then print them out.

I am using xelatex, and unicode text (but again, in this simple example spaces function as a delimiter).

Ideally, I would like to apply this wrapper to the entire document, right after \begin{document} and close right before \end{document}, so... it should be... robust... But... for simplicity, perhaps we may assume that it receives ordinary text, as an argument. Another idea is that maybe I will have to use a style file to do something like this?

Answer 1

Define dictionary words separated by || using the \setsepchar macro. Then, after \begin{document}, issue \def\currentword{}\tokencyclexpress and before \end{document}, issue \endtokencyclexpress. This sets up an environment where all the tokens of the document are screened in search of dictionary words first. If found, highlighting is added to the input token stream, prior to execution of the token list.

Note that word gets one style of highlighting if it is an exact match, and a different style if it merely contains the dictionary word, for example, plurals.

Highlight format for exact matches is set with \newcommand\autohighlightStyleA{\textcolor{red}}.

Highlight format for ``includes'' matches is set with \newcommand\autohighlightStyleB{\textcolor{blue}}.

If a dictionary word is broken across a group boundary, as in ap\textbf{ple}, it will not be recognized for highlight.

I should add that the approach can find dictionary words embedded in groups such as \textit{banana}, which is actually a difficult thing to accomplish, if it weren't for tokcycle.

\documentclass{article}
\usepackage{tokcycle,listofitems,xcolor}
\setsepchar{apple||banana}
\newcommand\testdict{%
  \if\relax\detokenize\expandafter{\currentword}\relax\else
    {\ignoreemptyitems
      \greadlist\dictcompA{\currentword}}%
    \readlist\dictcompB{\currentword}%
    \ifnum\listlen\dictcompA[]=0\relax
      \addcytoks[1]{\autohighlightStyleA}%
      \addcytoks[1]{\expandafter{\currentword}}
    \else
      \ifnum\listlen\dictcompB[]>1\relax
        \addcytoks[1]{\autohighlightStyleB}%
        \addcytoks[1]{\expandafter{\currentword}}
      \else
        \addcytoks[1]{\currentword}%
      \fi
    \fi
  \fi
  \gdef\currentword{}%
}
\makeatletter
\Characterdirective{\tctestifcatnx A#1{\g@addto@macro\currentword{#1}}
  {\testdict\addcytoks{#1}}}
\stripgroupingtrue
\Groupdirective{\testdict\groupedcytoks{\processtoks{#1}\testdict}}
\Macrodirective{\g@addto@macro\currentword{#1}}
\Spacedirective{\testdict\addcytoks{#1}}
\makeatother
\newcommand\autohighlightStyleA{\textcolor{red}}
\newcommand\autohighlightStyleB{\textcolor{blue}}
\begin{document}
\def\currentword{}\tokencyclexpress
\tableofcontents

\listoffigures

\section{My Section about a banana and two apples}

Monkeys like banana, but they don't like apple.

Monkeys like \textit{banana}, but \textit{they} don't like ap\textbf{ple}.

Monkeys \textit{like bananananas,} but \textit{they} don't like \textbf{apple}.
\begin{equation}
  \textrm{banana} = 2^\mathrm{apples}
\end{equation}
\subsection{Relevant figures about a banana}
\begin{figure}[ht]
\centering
\fbox{apple and banana}
\caption{A boxed version of ``apple and banana''}
\end{figure}
\endtokencyclexpress
\end{document}

p.s. This works in pdflatex, xelatex, and lualatex.