Line Breaking Question with Polyglossia

Question

I am working with multiple languages (with polyglossia and XeLaTeX), and I am having issues with extra line breaks where I don't expect them, also likely related to an issue of not hyphenating where it should. See the minimal example below. The line surprisingly breaks after the first word (זײַן) of every third entry, and puts a line-break instead of hyphenating at the discretionary hyphens. I believe that this has something to do with the \hfil in \entry, as suggested by this answer, but I don't want to get rid of \hfil (replacing it with \hspace{0.1cm plus 0.1cm}, for example), as I need both the left and right sides to be justified. Additionally, the last two entries show that it may also have to do with the interaction between the Yiddish text and the English text, as TeX hyphenates at the normal hyphen when there is no Yiddish, but it fails to hyphenate when there is Yiddish. The parbox at the very end of the sample code shows that \maq is a valid hyphenation spot, outside of my \entry environment, however after that I show that the hyphenation doesn't work properly in a normal paragraph, outside of \parbox.

I would like the right-hand side to fill the space before breaking (and in fact, I have manually edited the first three entries to appear as I expect all the entries to appear).

\documentclass[10pt]{book}
\usepackage{polyglossia}
\usepackage{multicol}
\usepackage{geometry}

\setdefaultlanguage{english}
\setotherlanguages{hebrew}
\newfontfamily\hebrewfont[Script=Hebrew]{Arial Unicode MS}

\newcommand\maq{\discretionary{־}{}{־}}
\newcommand{\h}[1]{\RL{\texthebrew{#1}\hfil}}

\newcommand{\hangparagentry}{\hangindent-0.5em \hangafter1 \leftskip 0.8em \parindent -0.5em}

\DeclareDocumentCommand{\entry}{ m o m }{\strut\noindent\hangparagentry{\hspace{-.5em}\textbf{#1}\hspace{0.1cm}\hfil \h{{#3}\IfNoValueF{#2}{ \textenglish{\scriptsize{#2}}}}\par}\strut}

\begin{document}

\setlength{\columnsep}{0.75pc}
\setlength{\columnseprule}{0.4pt}

\flushright

\begin{multicols}{2}

\entry{vehicle identification number}{דער אױטאָ־ אידענטיפֿיציר\maq נומער, ־ן}
\entry{vehicle inspection program}{די אױטאָ־ אינספּעקטיר\maq פּראָגראַם, ־ען}
\entry{be eager to be eager to}[[BALN] [KhÉYShEK] [LÓET/LÓER]]{זײַן אַ בעלן צו; האָבן \\ \hspace{0.5em} חשק צו; זײַן נײַגעריק ‹להוט› צוצו; האָבן  צו; זײַן נײַגעריק ‹להוט› צוצו;   צו; זײַן נײַגעריק ‹להוט› צו; זײ֜ער ‹שטאַרק› װעלן}
\entry{vehicle identification number}{דער אױטאָ\maq אידענטיפֿיציר\maq נומער, ־ן}
\entry{vehicle inspection program}{די אױטאָ\maq אינספּעקטיר\maq פּראָגראַם, ־ען}
\entry{be eager to be eager to}[[BALN] [KhÉYShEK] [LÓET/LÓER]]{זײַן אַ בעלן צו; האָבן חשק צו; זײַן נײַגעריק ‹להוט› צוצו; האָבן  צו; זײַן נײַגעריק ‹להוט› צוצו;   צו; זײַן נײַגעריק ‹להוט› צו; זײ֜ער ‹שטאַרק› װעלן}
\entry{vehicle identification number}{דער אױטאָ\maq אידענטיפֿיציר\maq נומער, ־ן}
\entry{vehicle inspection program}{די אױטאָ\maq אינספּעקטיר\maq פּראָגראַם, ־ען}
\entry{be eager to be eager to}[[BALN] [KhÉYShEK] [LÓET/LÓER]]{זײַן אַ בעלן צו; האָבן חשק צו; זײַן נײַגעריק ‹להוט› צוצו; האָבן  צו; זײַן נײַגעריק ‹להוט› צוצו;   צו; זײַן נײַגעריק ‹להוט› צו; זײ֜ער ‹שטאַרק› װעלן}
\entry{vehicle identification number}{דער אױטאָ\maq אידענטיפֿיציר\maq נומער, ־ן}
\entry{vehicle inspection program}{די אױטאָ\maq אינספּעקטיר\maq פּראָגראַם, ־ען}
\entry{be eager to be eager to}[[BALN] [KhÉYShEK] [LÓET/LÓER]]{זײַן אַ בעלן צו; האָבן חשק צו; זײַן נײַגעריק ‹להוט› צוצו; האָבן  צו; זײַן נײַגעריק ‹להוט› צוצו;   צו; זײַן נײַגעריק ‹להוט› צו; זײ֜ער ‹שטאַרק› װעלן}
\entry{vehicle identification number}{די \textenglish{tests-testing}}
\entry{vehicle identification number}{\textenglish{tests-longtestings}}

\end{multicols}

\parbox{0pt}{\h{דער אױטאָ\maq אידענטיפֿיציר\maq נומער, ־ן}}

\h{די אױטאָ\maq אינספּעקטיר\maq פּראָגראַם, ־ען די אױטאָ\maq אינספּעקטיר\maq פּראָגראַם, ־ען די אױטאָ\maq אינספּעקטיר\maq פּראָגראַם, ־ען די אױטאָ\maq אינספּעקטיר\maq פּראָגראַם, ־ען די אױטאָ\maq אינספּעקטיר\maq פּראָגראַם, ־ען די אױטאָ\maq אינספּעקטיר\maq פּראָגראַם, ־ען}

\end{document}

Answer 1

Your problem is certainly coming from the \hfil, but you can solve your issue by telling TeX that you want to look for solutions that involve hyphens.

Basically, when TeX builds a paragraph, the line breaking algorithm puts a penalty on every space and every possible hyphen, and then determines which will be the least costly (penalty wise).

You can solve this by putting this

\hyphenpenalty=-500

Somewhere in your preamble. Or even in your definition of entry, if you don't want to encourage hyphens in the rest of your text.

Negative penalties encourage the hyphens, and I tried your document and this solves the issue. Observe:

Answer 2

This question has been bugging me for a while. Here is my best effort. It's not as good as A Gold Man's though (which is new since I started thinking about this).

You can break at the maqqef by specifying HyphenChar=־ for your Hebrew font.

But the \hfil in your definition is still a problem. I solved it by putting the entry into a \vbox with a calculated \parshape and then unboxing this to collapse the first line. Then you can insert the an \hfill.

There is a downside though. You can't then break an entry across a page or column. You might be able to solve this for the page break, but I think definitely not for the column break as even \tcolorbox can't do this.

Anyway, here's the MWE:

\documentclass{article}

\usepackage{geometry}
\usepackage{multicol}
\usepackage{polyglossia}
\setdefaultlanguage{english}
\setotherlanguage{hebrew}

\newfontfamily\hebrewfont{SBL BibLit}[Script=Hebrew,HyphenChar={־}]

\parindent=0em

\newlength\entrywidth
\newlength\entrysep
\newlength\entryindent
\entrysep=1em
\entryindent=1em

% \shapeentry adapted from Victor Eijkhout, TeX by Topic.
\newcounter{entryline}
\newbox\linebox \newbox\snapbox
\def\shapeentry{%
  \ifnum\value{entryline}=0
    \leftskip 0pt
    \rightskip 0pt plus 1fil
  \fi
  \setbox\linebox\lastbox
  \ifvoid
    \linebox
  \else
    \unskip\unpenalty
    {\shapeentry}%
    \stepcounter{entryline}%
    \setbox\snapbox\hbox{\unhcopy\linebox}%
    \ifnum\value{entryline}=1
      \mbox{\box\snapbox}\hfill\entryname\break
    \else
      \box\linebox\hskip\entryindent\break
    \fi
  \fi}

\DeclareDocumentCommand{\entry}{ m o m }{%
  \parindent=0em
  \setcounter{entryline}{0}
  \gdef\entryname{\textenglish{\textbf{#1}}}
  \settowidth{\entrywidth}{\entryname\hskip\entrysep}
  \begin{hebrew}
    \vbox{%
      \parshape=2
                \entrywidth \dimexpr\linewidth-\entrywidth
                \entryindent \dimexpr\linewidth-2\entryindent
      #3\IfNoValueF{#2}{\space\textenglish{\scriptsize #2}}\par\shapeentry}
  \end{hebrew}
  \vskip-\baselineskip
}

\setlength{\columnsep}{0.75pc}
\setlength{\columnseprule}{0.4pt}
\pagestyle{empty}

\begin{document}

\begin{multicols}{2}
  \entry{vehicle identification number}{דער אױטאָ־אידענטיפֿיציר־נומער, ־ן}
  \entry{vehicle inspection program}{די אױטאָ־אינספּעקטיר־פּראָגראַם, ־ען}
  \entry{be eager to be eager to}[[BALN] [KhÉYShEK] [LÓET/LÓER]]{זײַן אַ בעלן
    צו; האָבן חשק צו; זײַן נײַגעריק ‹להוט› צוצו; האָבן צו; זײַן נײַגעריק ‹להוט› צוצו;
  צו; זײַן נײַגעריק ‹להוט› צו; זײ֜ער ‹שטאַרק› װעלן}
  \entry{vehicle identification number}{דער אױטאָ־אידענטיפֿיציר־נומער, ־ן}
  \entry{vehicle inspection program}{די אױטאָ־אינספּעקטיר־פּראָגראַם, ־ען}
  \entry{be eager to be eager to}[[BALN] [KhÉYShEK] [LÓET/LÓER]]{זײַן אַ בעלן
    צו; האָבן חשק צו; זײַן נײַגעריק ‹להוט› צוצו; האָבן צו; זײַן נײַגעריק ‹להוט› צוצו;
  צו; זײַן נײַגעריק ‹להוט› צו; זײ֜ער ‹שטאַרק› װעלן}
  \entry{vehicle identification number}{דער אױטאָ־אידענטיפֿיציר־נומער, ־ן}
  \entry{vehicle inspection program}{די אױטאָ־אינספּעקטיר־פּראָגראַם, ־ען}
  \entry{be eager to be eager to}[[BALN] [KhÉYShEK] [LÓET/LÓER]]{זײַן אַ בעלן
    צו; האָבן חשק צו; זײַן נײַגעריק ‹להוט› צוצו; האָבן  צו; זײַן נײַגעריק ‹להוט› צוצו;
    צו; זײַן נײַגעריק ‹להוט› צו; זײ֜ער ‹שטאַרק› װעלן}
  \entry{vehicle identification number}{דער אױטאָ־אידענטיפֿיציר־נומער, ־ן}
  \entry{vehicle inspection program}{די אױטאָ־אינספּעקטיר־פּראָגראַם, ־ען}
  \entry{be eager to be eager to}[[BALN] [KhÉYShEK] [LÓET/LÓER]]{זײַן אַ בעלן
    צו; האָבן חשק צו; זײַן נײַגעריק ‹להוט› צוצו; האָבן  צו; זײַן נײַגעריק ‹להוט› צוצו;
    צו; זײַן נײַגעריק ‹להוט› צו; זײ֜ער ‹שטאַרק› װעלן}
\end{multicols}
\end{document}

Answer 3

Perhaps I'm not understanding your \entry definition, but I suspect the problem is that you aren't telling latex which parts of your text are in Hebrew, so it isn't applying the correct hyphenation rules. For example, I am currently writing a multi-lingual two column document (facing translation) with paracol and polyglossia and I only get the proper hyphenation behavior if I explicitly declare the other (non-english) languages (english is the default, like in your example). For example:

\documentclass[letterpaper,titlepage,twoside,openany,12pt]{book}

\usepackage{paracol} 
%more space between the columns to trigger more hyphenation
\setlength{\columnsep}{90 pt} 

\usepackage{polyglossia}
\setdefaultlanguage[variant=american]{english}
\setotherlanguages{icelandic,danish,norsk,swedish}

%encourage hyphenation for this example
\hyphenpenalty=0
\tolerance=900
\setlength{\emergencystretch}{0.2em}

\begin{document}

\begin{paracol}{2}

\switchcolumn[0]*
\noindent
Once upon a time, in a kingdom far far away (so far away that no one knows
its name) there was a king who had an only child, a daughter.  She was so
sullen and gloomy that no one had ever seen her laugh.  She was a grown
woman, very pretty and kind; but, she was always sad and never happy ---
either she cried or she sulked.  It was simply impossible to make her
smile.

\switchcolumn
\begin{danish}
\noindent
Der var engang en Konge i et Land langt langt borte, som der er ingen, der
ved Navn på; han havde kun en eneste Datter.  Hun var så sum og så sur,
at der var aldrig nogen, der havde set hende le.  Hun var nu en voksen Pige
og en køn Pige og en rar Pige, men altid var hun sorrigfuld og aldrig glad;
enten græd hun eller også surmulede hun.  Det var ikke til at slå et Smil
af hende med en Vognkæp.
\unskip
\end{danish}

\end{paracol}

\end{document}

(Notice all the hyphenated danish words in the right-hand column).

Note: I get unwanted spaces and line breaks unless I include an \unskip before the \end{language} tag.