soul package command \hl can not work with escaped space

It seems that \hl command in the soul package can not work with escaped space. A minimal non-working example is

\documentclass{article}
\usepackage{xcolor, soul}

\colorlet{mycolor}{red!30}
\sethlcolor{mycolor}

\begin{document}

\hl{can\ we\ highlight}

\end{document}


The error message is

Package soul Error: Reconstruction failed.

Some posts (for example, here and here) suggest using \mbox(or \hbox) to wrap the text or using \protect. While these methods work, they have some drawbacks:

• If \mbox is used, long text will not wrap normally.
• If \protect is used, the background color for text will disappear.

How to solve this problem more elegantly?

Edit

The solution shared by Skillmon works for English, but not for Chinese. The following MWE will illustrate that.

\documentclass{ctexart}
\usepackage{xcolor, soul}
\usepackage{xparse}

\let\hlORIG\hl

\colorlet{mycolor}{red!30}
\sethlcolor{mycolor}

\ExplSyntaxOn
\tl_new:N \l_jdhao_hlx_tl
\RenewDocumentCommand \hl { +m }
{
\tl_set:Nn \l_jdhao_hlx_tl { #1 }
\tl_replace_all:Nnn \l_jdhao_hlx_tl { \  } { ~ }
\exp_args:NV \hlORIG \l_jdhao_hlx_tl
}
\ExplSyntaxOff

\begin{document}

\hl{test\ test}  %% pass!
\hl{测试\ 测试} %% fail!

\end{document}


It is interesting that English test passed, but the Chinese test failed. The error message is the same.

• Out of interest: Why do you want to escape spaces after normal words? – TeXnician Sep 4 '18 at 10:30
• Actually, I was trying to convert Markdown to PDF using Pandoc. For space in inline code, Pandoc will escape it, which unfortunately conflicts with soul. – jdhao Sep 4 '18 at 10:32

With a helper macro which converts all the explicit spaces (\) into a normal space, this is possible (I redefined the \hl macro so that you don't have to change anything else):

\documentclass{article}
\usepackage{xcolor, soul}
\usepackage{xparse}
\let\hlORIG\hl

\colorlet{mycolor}{red!30}
\sethlcolor{mycolor}

\ExplSyntaxOn
\tl_new:N \l_jdhao_hlx_tl
\RenewDocumentCommand \hl { +m }
{
\tl_set:Nn \l_jdhao_hlx_tl { #1 }
\tl_replace_all:Nnn \l_jdhao_hlx_tl { \  } { ~ }
\exp_args:NV \hlORIG \l_jdhao_hlx_tl
}
\ExplSyntaxOff

\begin{document}

\hl{can\ we\ highlight}

\end{document}


EDIT: Working with Chinese input, but breaking automatic hyphenation (you can still provide manual hyphenation with \-):

\documentclass{ctexart}
\usepackage{xcolor, soul}
\usepackage{xparse}
%\usepackage[chinese,main=english]{babel}
\let\hlORIG\hl

\colorlet{mycolor}{red!30}
\sethlcolor{mycolor}

\ExplSyntaxOn
\tl_new:N \l_jdhao_hlx_tl
\RenewDocumentCommand \hl { m }
{
\tl_set:Nn \l_jdhao_hlx_tl { #1 }
\tl_replace_all:Nnn \l_jdhao_hlx_tl { \  } { ~ }
\regex_replace_all:nnN { [^\s\c{-}] } { \c{hbox} \0 } \l_jdhao_hlx_tl
\exp_args:NV \hlORIG \l_jdhao_hlx_tl
}
\ExplSyntaxOff

\begin{document}
% for comparison of hyphenating
\rule{.9\textwidth}{1pt}possibility\ to\ highlight

\rule{.9\textwidth}{1pt}\hl{possibility\ to\ highlight}

\rule{.9\textwidth}{1pt}\hl{pos\-sibility\ to\ highlight}

\rule{.9\textwidth}{1pt}\hlORIG{possibility to highlight}

\hl{汤面}
\hl{汤面}
\hl{测试\ 测试}
%\hlORIG{汤面}

\end{document}


EDIT2:

Yet another version which allows you to use braces and other stuff in the argument of \hl. Note that both this version and the second version break maths in the argument of \hl. You can change the regular expression used as the first argument of \regex_replace_all:nnN to fix more issues you're facing, I'm not willing to put more time into fiddling with it.

\documentclass{ctexart}
\usepackage{xcolor, soul}
\usepackage{xparse}
%\usepackage[chinese,main=english]{babel}
\let\hlORIG\hl

\colorlet{mycolor}{red!30}
\sethlcolor{mycolor}

\ExplSyntaxOn
\tl_new:N \l_jdhao_hlx_tl
\RenewDocumentCommand \hl { m }
{
\tl_set:Nn \l_jdhao_hlx_tl { #1 }
\tl_replace_all:Nnn \l_jdhao_hlx_tl { \  } { ~ }
\regex_replace_all:nnN { \c[^CBEMTPUDA]\S } { \c{hbox} \0 } \l_jdhao_hlx_tl
\exp_args:NV \hlORIG \l_jdhao_hlx_tl
}
\ExplSyntaxOff

\begin{document}
% for comparison of hyphenating
\rule{.9\textwidth}{1pt}possibility\ to\ highlight

\rule{.9\textwidth}{1pt}\hl{possibility\ to\ highlight}

\rule{.9\textwidth}{1pt}\hl{pos\-sibility\ \textbackslash{} to\ highlight}

\rule{.9\textwidth}{1pt}\hlORIG{possibility\textbackslash{} to highlight}

\hl{汤面}
\hl{汤面}
\hl{测试\ 测试}

\end{document}

• It seems that this solution does not work for Chinese text, it only works for English. I have tried to test it with Chinese text and failed. – jdhao Sep 4 '18 at 13:10
• @jdhao can you please update your question then, to include Chinese as well? This way any future answerer knows that this has to work for Chinese. And please also add information which engine you use to compile (there might be a clever solution for LuaTeX if you use it). – Skillmon Sep 4 '18 at 13:59
• @jdhao I can add code which works with chinese text, but breaks hyphenation (line breaking still works, though). – Skillmon Sep 4 '18 at 17:11
• I have updated the question to include the fail case for Chinese. – jdhao Sep 5 '18 at 2:44
• @jdhao my updated answer does work with your updated MWE (though breaking hyphenation -- but I don't know how hyphenation works in Chinese anyway). – Skillmon Sep 5 '18 at 6:45