# \tl_replace_all:Nnn breaking with certain unicode characters?

Certain sequences of Unicode characters, particularly involving Hebrew yud (י) seem to have strange behavior with \tl_replace_all:Nnn. In the following MWE for XeLaTeX, some of these sequences when used as the search string create unexplained indentation even with no matches while others do not (almost all other Hebrew sequences work as expected). I also saw identical behavior for the same sequences with \regex_replace_all:nnN though I do not include it here. Any thoughts?

\documentclass{article}
\usepackage{xparse}
\usepackage{expl3}
\usepackage{polyglossia}
\setdefaultlanguage{hebrew}
\newfontfamily\hebrewfont[Script=Hebrew]{Taamey Frank CLM}

\ExplSyntaxOn
\NewDocumentCommand{\transliterateA}{m}
{
\hebrew_replaceA:n {#1}
}
\NewDocumentCommand{\transliterateB}{m}
{
\hebrew_replaceB:n {#1}
}
\tl_new:N \l_hebrew_input_text_tl
\cs_new_protected:Npn \hebrew_replaceA:n #1
{
\tl_set:Nn \l_hebrew_input_text_tl { #1 }
%handle yud-vowel before it accidentally gets turned into an i
\tl_replace_all:Nnn \l_hebrew_input_text_tl { יְ } { yְ }
\tl_replace_all:Nnn \l_hebrew_input_text_tl { יֻ } { yֻ }
%handle yud-dagesh-vowel before it accidentally gets turned into an i
\tl_replace_all:Nnn \l_hebrew_input_text_tl { יְּ } { yyְ }
\tl_replace_all:Nnn \l_hebrew_input_text_tl { יֻּ } { yyֻ }
\tl_use:N \l_hebrew_input_text_tl
}
\cs_new_protected:Npn \hebrew_replaceB:n #1
{
\tl_set:Nn \l_hebrew_input_text_tl { #1 }
%handle yud-vowel before it accidentally gets turned into an i
\tl_replace_all:Nnn \l_hebrew_input_text_tl { יִ } { yִ }
\tl_replace_all:Nnn \l_hebrew_input_text_tl { יֱ } { yֱ }
%handle yud-dagesh-vowel before it accidentally gets turned into an i
\tl_replace_all:Nnn \l_hebrew_input_text_tl { יִּ } { yyִ }
\tl_replace_all:Nnn \l_hebrew_input_text_tl { יֱּ } { yyֱ }
\tl_use:N \l_hebrew_input_text_tl
}
\ExplSyntaxOff

\begin{document}
\transliterateA{שְֿׁ שְֿׁ שְֿׁ שְֿׁ שְֿׁ שְֿׁ שְּֿׁ שְּֿׁ שְּֿׁ שְּֿׁ}

\transliterateB{שְֿׁ שְֿׁ שְֿׁ שְֿׁ שְֿׁ שְֿׁ שְּֿׁ שְּֿׁ שְּֿׁ שְּֿׁ}
\end{document}


• If I compile your code I get both lines with the same indentation... That shouldn't be caused by \tl_replace_all:Nnn though, as it doesn't add typesetting instructions like indentation. That said, you can examine the contents of the token list with \tl_analysis_show:N \l_hebrew_input_text_tl May 13, 2020 at 1:54
• Hmmm! I used \tl_analysis_show:N and the results are identical for both cases. But, inspired by this, I had a look at the raw characters of each line. The offending lines begin with two U+00A0 : NO-BREAK SPACE [NBSP] rather than ordinary spaces. Switching this fixes the problem. I wonder if the NBSPs are somehow sanitized on this forum. Thanks! May 13, 2020 at 2:18
• Yeah, If I put a non-breaking space at the beginning of one of the lines it gets shifted (though not nearly as much as in your picture), because TeX won't discard that at the beginning of a line. And yes, this site filters a lot of characters. Glad your problem's solved! It would be nice if you wrote an answer for future readers :-) May 13, 2020 at 2:29