2

Certain sequences of Unicode characters, particularly involving Hebrew yud (י) seem to have strange behavior with \tl_replace_all:Nnn. In the following MWE for XeLaTeX, some of these sequences when used as the search string create unexplained indentation even with no matches while others do not (almost all other Hebrew sequences work as expected). I also saw identical behavior for the same sequences with \regex_replace_all:nnN though I do not include it here. Any thoughts?

\documentclass{article}
\usepackage{xparse}
\usepackage{expl3}
\usepackage{polyglossia}
\setdefaultlanguage{hebrew}
\newfontfamily\hebrewfont[Script=Hebrew]{Taamey Frank CLM}

\ExplSyntaxOn
\NewDocumentCommand{\transliterateA}{m}
 {
  \hebrew_replaceA:n {#1}
 }
 \NewDocumentCommand{\transliterateB}{m}
 {
  \hebrew_replaceB:n {#1}
 }
\tl_new:N \l_hebrew_input_text_tl
\cs_new_protected:Npn \hebrew_replaceA:n #1
 {
  \tl_set:Nn \l_hebrew_input_text_tl { #1 }
  %handle yud-vowel before it accidentally gets turned into an i
  \tl_replace_all:Nnn \l_hebrew_input_text_tl { יְ } { yְ }
  \tl_replace_all:Nnn \l_hebrew_input_text_tl { יֻ } { yֻ }
  %handle yud-dagesh-vowel before it accidentally gets turned into an i
  \tl_replace_all:Nnn \l_hebrew_input_text_tl { יְּ } { yyְ }
  \tl_replace_all:Nnn \l_hebrew_input_text_tl { יֻּ } { yyֻ }
  \tl_use:N \l_hebrew_input_text_tl
 }
 \cs_new_protected:Npn \hebrew_replaceB:n #1
 {
  \tl_set:Nn \l_hebrew_input_text_tl { #1 }
  %handle yud-vowel before it accidentally gets turned into an i
  \tl_replace_all:Nnn \l_hebrew_input_text_tl { יִ } { yִ }
  \tl_replace_all:Nnn \l_hebrew_input_text_tl { יֱ } { yֱ }
  %handle yud-dagesh-vowel before it accidentally gets turned into an i
  \tl_replace_all:Nnn \l_hebrew_input_text_tl { יִּ } { yyִ }
  \tl_replace_all:Nnn \l_hebrew_input_text_tl { יֱּ } { yyֱ }
  \tl_use:N \l_hebrew_input_text_tl
 }
\ExplSyntaxOff

\begin{document}
\transliterateA{שְֿׁ שְֿׁ שְֿׁ שְֿׁ שְֿׁ שְֿׁ שְּֿׁ שְּֿׁ שְּֿׁ שְּֿׁ}

\transliterateB{שְֿׁ שְֿׁ שְֿׁ שְֿׁ שְֿׁ שְֿׁ שְּֿׁ שְּֿׁ שְּֿׁ שְּֿׁ}
\end{document}

enter image description here

Improve this question
3
  • 2
    If I compile your code I get both lines with the same indentation... That shouldn't be caused by \tl_replace_all:Nnn though, as it doesn't add typesetting instructions like indentation. That said, you can examine the contents of the token list with \tl_analysis_show:N \l_hebrew_input_text_tl
    – Phelype Oleinik
    May 13, 2020 at 1:54
  • Hmmm! I used \tl_analysis_show:N and the results are identical for both cases. But, inspired by this, I had a look at the raw characters of each line. The offending lines begin with two U+00A0 : NO-BREAK SPACE [NBSP] rather than ordinary spaces. Switching this fixes the problem. I wonder if the NBSPs are somehow sanitized on this forum. Thanks!
    – sideband
    May 13, 2020 at 2:18
  • 1
    Yeah, If I put a non-breaking space at the beginning of one of the lines it gets shifted (though not nearly as much as in your picture), because TeX won't discard that at the beginning of a line. And yes, this site filters a lot of characters. Glad your problem's solved! It would be nice if you wrote an answer for future readers :-)
    – Phelype Oleinik
    May 13, 2020 at 2:29

1 Answer 1

Reset to default
1

This was actually a PEBKAC issue.

The offending lines begin with two U+00A0: NO-BREAK SPACE (which, it turns out, can accidentally be typed by alt-shift-space in some keyboard layouts) rather than ordinary spaces. Switching to ordinary spaces fixes the problem.

Additionally, the reason Phelype was not able to recreate the issue was that StackExchange sanitizes NBSPs into ordinary spaces, so my posted MWE then worked properly. Thanks, Phelype.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .