# \noindent has no effect when surrounded by bidi markers

Why does xelatex indent the first line of (Hebrew) text in the pdf obtained from the following LaTeX manuscript?

\documentclass{scrartcl}

\catcode8198=10 % Left-To-Right Mark
\catcode8199=10 % Right-To-Left Mark
\catcode8234=10 % Left-To-Right Embedding
\catcode8235=10 % Right-To-Left Embedding
\catcode8236=10 % Pop Directional Formatting

\usepackage{fontspec}
\setmainfont{Arial Hebrew}
\usepackage[rldocument]{bidi}

\begin{document}
\setRTL

‏‪\noindent‬ פְּרֶצְל מנסה לישון, על מנת להחיש את מהלך הזמן. אדם הולך לו לישון בשתים-עשרה, קם כעבור רגע, והנה השעה שבע בבוקר, ומתברר שהוא השתמש בשינה בתור סוס, לדלג על שבע שעות. תחבולה זו ידועה היטב לטבע. מה עושה הטבע נגדה? נותן לפרש הערמומי להירדם בשתים-עשרה, מציף אותו בהרבה סיוטים, וכשהפרש קם תשוש ומזיע ואומר לעצמו בחצי סיפוק "לפחות עברו שבע שעות," מייד קופץ שעונו ומורה שתים-עשרה ורבע. גם לא ישנת, בסיוטים סוייטת, וכל הלילה עוד לפניך.

\end{document}


The typeset output:

A note on invisible bidi (bidirectionality) markers

The \noindent appears before all the Hebrew text. The \noindent is surrounded by bidi markers. It is immediately preceded by two markers: Unicode Right-To-Left Mark (U+200F) and Left-To-Right Embedding (U+202A), in this order; and immediately followed by a Pop Directional Formatting (U+202C) marker. So the character order is, form left to right:

U+200F, U+202A, \, n, o, i, n, d, e, n, t, U+202C, U+0020 (a space), <Hebrew text>

To help visualize this, avail yourselves of the following illustration.

Legend
⊳ ... Right-To-Left Mark, U+200F (typeset as ⊲ inside a rtl paragraph)
⋉ ... Left-To-Right Embedding, U+202A
⋈ ... Pop Directional Formatting, U+202C

Each of these symbols was typeset in the editor window after the instruction it represents has taken effect.

Your problem is putting U+200F at the start of your paragraph. That is a catcode 12 token: 'other', and so starts a paragraph, triggering the insertion of the paragraph indent and \everypar. The fact that it's non-printing makes no difference to TeX. The \noindent therefore occurs after the paragraph has started, and does nothing at all.

(The catcode setting at the start of the document is likely not as intended as 8198 = U+2006 = SIX-PER-EM SPACE and 8199 = U+2007 = FIGURE SPACE.)

• @EvanAad I'll discuss with the team whether more of the \catcode values in this area can be set automatically – Joseph Wright Jun 29 '17 at 17:05
• Yes, please! It only makes sense (to me) that all the bidi markers covered in the Unicode standard (Annex #9 'Unicode Bidirectional Algorithm', section 4.2), namely ALM, RLM, LRM, LRE, RLE, LRO, RLO, PDF, FSI, LRI, RLI, and PDI, have catcode 10. – Evan Aad Jun 29 '17 at 17:18
• @EvanAad they would not work at all as catcode 10 as they would be tokenised as character 32 – David Carlisle Jun 29 '17 at 18:24
• @DavidCarlisle: This would be acceptable. As I see it, these characters are intended for the editor application; TeX has its own markup to deal with these issues. The only problem I can think of is what happens if one of these bidi marks is followed by a punctuation mark which should be typeset flush against the preceding word. – Evan Aad Jun 29 '17 at 18:53
• @EvanAad if you want to ignore the controls and only use markup, I think catcode 9 would be better than 10 (that is "ignore" rather than "space") – David Carlisle Jun 29 '17 at 19:04

It is not, in general correct to give the Unicode directional controls catcode 10 as then they can not control the formatting and will produce anomalous space.

Consider the string

abc‮xyz‬123


That is

  U+0061 LATIN SMALL LETTER A     a
U+0062 LATIN SMALL LETTER B     b
U+0063 LATIN SMALL LETTER C     c
U+202e RIGHT-TO-LEFT OVERRIDE
U+0078 LATIN SMALL LETTER X     x
U+0079 LATIN SMALL LETTER Y     y
U+007a LATIN SMALL LETTER Z     z
U+202c POP DIRECTIONAL FORMATTING
U+0031 DIGIT ONE     1
U+0032 DIGIT TWO     2
U+0033 DIGIT THREE     3


If set with the default catcodes (12= "other") then

\documentclass{article}
\usepackage{fontspec}
\setmainfont{Arial}
%\catcode"202E=10
%\catcode"202C=10
\begin{document}

abc‮xyz‬123
\end{document}


produces

Which is the specified order, matching the display in the browser, but if you uncoment the declarations to make the controls catcode 10 then you get

with no direction change, and spurious space.

• I guess we are back here with the question of whether such demo behaviour with chars which are LTR is correct anyway (yes, it's marked up as RTL by y, z, z are all Latin chars so should be LTR) – Joseph Wright Jun 29 '17 at 19:01
• @JosephWright nope that's what U+202E means it's override so forces RTL, but I only used that so I could type latin example, I could have used naturally rtl characters and simple rtl mark and the anomolous spaces would still have shown up if the controls were catcode 10 – David Carlisle Jun 29 '17 at 19:06
• I mean 'Unicode may have invented an override, but it fundamentally doesn't make sense for Latin characters'. If you want to write zyx not xyz, that's exactly what you do. Direction is a feature of the writing system to which characters belong ... – Joseph Wright Jun 29 '17 at 19:07
• @JosephWright it doesn't make sense for any characters as it forces an unnatural order but that's what it's specified to do but as I say that's just here to make it easier to read the example if (like me) you can't read Hebrew. instead of xyz I could have used hebrew characters and U+200F instead of 202E and the bad spaces would still have shown up – David Carlisle Jun 29 '17 at 19:11
• @DavidCarlisle: One doesn't need to understand Hebrew to type Hebrew letters or to copy a paragraph from the Internet, as I did in my post. Anyone who's used social media or read the papers knows that it's possible to write profusely without making sense. At any rate, thanks for your sound advice re catcode 12. – Evan Aad Jun 29 '17 at 19:56