wordcount with xesearch errors

Question

I'm using some code I found here on stackexchange to count words in a LaTeX document. I'm using TeXLive on a Mac and using XeLateX to produce the pdf document. It produces a pdf, but with an error message, and longer documents fail to produce the correct pdf.

Here's a minimal document:

\documentclass{article}

\usepackage{xesearch}
\newcounter{words}
\newenvironment{wordcount}{%
\setcounter{words}{0}
\SearchList!{wordcount}{\stepcounter{words}}
    {a?,b?,c?,d?,e?,f?,g?,h?,i?,j?,k?,l?,m?,
    n?,o?,p?,q?,r?,s?,t?,u?,v?,w?,x?,y?,z?}
\UndoBoundary{'’-}
\MakeBoundary{„“‚‘–}
\SearchOrder{p;}}{%
\StopSearching}

\begin{document}
\begin{wordcount}

Here are some words. 
Here are some more words.

\end{wordcount}
\end{document}

Here's the error message:

! Undefined control sequence.
<XeTeXinterchartoks> \xdef \xs@String {\xs@String
e}
l.20 Her
e are some words.
The control sequence at the end of the top line
of your error message was never \def'ed. If you have
misspelled it (e.g., `\hobx'), type `I' and the correct
spelling (e.g., `I\hbox'). Otherwise just continue,
and I'll forget about whatever was undefined.

I have XeTeX version 3.14159265-2.6-0.99996 (TeX Live 2016), and the most recent xesearch version available in the TeXLive utility (0.1). This worked fine when I was using Tex Live 2014 version, but so many things have been updated now I don't know where to start figuring out what the problem is.

Answer 1

The latest version of XeTeX has increased the number of character classes to 4096, but xesearch still assumes the upper bound is 255 instead of 4095.

You have to redo the initializations.

\documentclass{article}

\usepackage{xesearch}

\makeatletter
\chardef\xs@NatDel=\e@alloc@intercharclass@top
\chardef\xs@lrDel=\numexpr\e@alloc@intercharclass@top-1\relax
\chardef\xs@Classes=\numexpr\e@alloc@intercharclass@top-2\relax
\XeTeXinterchartoks\xs@lrDel\xs@Classless={\xs@LearnLetter}
\XeTeXinterchartoks\xs@NatDel\xs@Classless={\xs@LearnLetter}
\XeTeXinterchartoks\xs@NatDel\xs@lrDel{\xs@EndString}
\xs@MakeDel\{\}.,;:!?[()]-'`\xs@end
\makeatother

\newcounter{words}
\newenvironment{wordcount}{%
\setcounter{words}{0}
\SearchList!{wordcount}{\stepcounter{words}}
    {a?,b?,c?,d?,e?,f?,g?,h?,i?,j?,k?,l?,m?,
    n?,o?,p?,q?,r?,s?,t?,u?,v?,w?,x?,y?,z?}
\UndoBoundary{'’-}
\MakeBoundary{„“‚‘–}
\SearchOrder{p;}}{%
\StopSearching}

\begin{document}
\begin{wordcount}

Here are some words. 
Here are some more words.

\end{wordcount}

Number of words: \thewords

\end{document}

If you find \e@alloc@intercharclass@top to be undefined, update your TeX distribution.