It doesn't hyphenate words ending with “—”

Question

UPDATE

Well. After a while, I decided to see again about this “problem”, and just discovered that in pdfLaTeX, using \usepackage[utf8]{inputenc} and the — (the unicode —, not the ligature ---) it works perfect (at least in what I tried). Minor edit: as @cfr mentions in her answer (I forgot it), it is in fact possible to use --- (the ligature) if you use T1 encoding and \hyphenchar\font=\string"7F. In both methods (wether I use the em-dash — or ---) microtype works perfectly.

Now, the problem remains in XeLaTeX. I would like to clear my idea: I want the em-dash to behave correctly; it doesn't matter if I have to type — (the unicode em-dash —) or --- (the usual LaTeX way); it should (among others) hyphenate words correctly; be always together to the word; in case it's followed by a comma/period, they should be together; and, microtype should work, e.g., the hyphen should still hang in the margin.

Of course, there is a basic solution (which works in any engine): indicate to XeLaTeX the breaking points, for instance ocur\-recone\-stedoc\-umento---. But I'm looking for an automatic solution.

If you have anything to say, please, say it, it's welcomed!

---

Note: Every — you see in the code is an em-dash.

I'writing a paper, and XeLaTeX doesn't hyphenate words which end with – (traditional LaTeX ---). After reading the comments, I realized this is a common problem also in LaTeX (not only XeLaTeX). Here it is a minimal working example:

This code (full example at the bottom of the question) outputs

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento— con algunas palabras más.

If we substitute the last — by a comma, for example it hyphenates correctly

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento, con algunas palabras más.

Moreover, if the phrase ends with —. XeLaTeX (or whoever is doing this) takes the full stop to the next line

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento—.

Here is a full minimal working example

\documentclass{scrartcl}
\usepackage[hmargin = 4cm]{geometry}
\usepackage{fontspec}

\begin{document}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento— con algunas palabras más.

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
\end{document}

Any ideas?

Answer 1

You can prepare the following example:

\input ucode
\input lmfonts

\hsize=12cm

—Hola, esto es un texto absurdo —para ejemplificar lo que
ocurreconestedocumento— con algunas palabras más.

\end

and you can try to process it by 1) xetex test and 2) xetex -fmt pdfcsplain test. You will see different results: 1) the long word isn't hyphenated, 2) the long word is hyphenated.

You can study the difference in code settings, hyphenation settings etc. IMHO LuaLaTeX copies the settings from xetex's eplain, no from pdfcsplain. So, you have the problem.

Edit: Where is the difference? We can see the following setting in xetex.ini and in xelatex.ini:

 \XeTeXdashbreakstate=1

Bingo! Here is the problem. Set \XeTeXdashbreakstate=0 and your words will be hyphenated.

Answer 2

You can read an entry about this problem in my (now abandoned) blog, why it happens, and how it is solved in spanish babel.

But you are using xelatex and polyglossia, and I don't know if some solution is already included in this page. Anyway, it is easy to adapt the ideas and techniques used by babel, and define the following command:

\def\raya{%
\nobreak\hskip0pt\hbox{---}\nobreak\hskip0pt%
}

You have to put \raya{} instead of ---. So, in your example:

\raya{}Hola, esto es un texto absurdo \raya{}para ejemplificar lo que ocurreconestedocumento\raya{} con algunas palabras m
ás.

\raya{}Hola, esto es un texto absurdo \raya{}para ejemplificar lo que ocurreconestedocumento\raya{}, con algunas palabras 
más.

\raya{}Hola, esto es un texto absurdo \raya{}para ejemplificar lo que ocurreconestedocumento\raya{}.

And this is the output:

Update

As requested by the OP,it is possible to make unicode character — active (this is easy in xelatex, since it has native utf8 input), to define — as a new Unicode char, and then use — instead of \raya [Thanks to egreg for pointing me to package newunicodechar, which is a cleaner solution than my previous attempt changing catcodes, and does not have issues with spaces after the character]:

\documentclass{scrartcl}
\usepackage[hmargin = 4cm]{geometry}
\usepackage{fontspec}
\usepackage{newunicodechar}

\newunicodechar—{%
\leavevmode\nobreak\hskip0pt\hbox{---}\nobreak\hskip0pt\relax%
}

\begin{document}
—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento— con algunas palabras más.

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento—, con algunas palabras más.

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento—.
\end{document}

Resulting in:

Answer 3

In regular TeX with the T1 encoding there is a way to avoid this problem which involves redefining the hyphen character. This is possible because T1 encodes two hyphen characters which look identical (in most fonts) but which play different roles.

The beauty of this is that it does not require the use of any special commands in the way the babel solution does. All you do is add a single line of code following \begin{document}:

\hyphenchar\font=\string"7F

after loading T1 with the fontenc package. What this does is tells TeX to use the character is slot 127 as the hyphenation character. That is, when TeX needs to break a word across lines, it will use "7F to hyphenate the word. It does not change the character you get when you type '-', however. That character corresponds to the one in slot 45 of the T1 encoding. So TeX does not see a word which is already hyphenated as hyphenated. Hence the prohibition on hyphenating already hyphenated words does not apply, and TeX breaks the word as appropriate. This also retains ligaturing since it is the character in slot 45 - not the one in slot 127 - which is defined in ligatures such as '--' and '---' in T1. So you can break the norms of English typesetting with impunity!

So I wondered if something similar might be possible with XeLaTeX as well. The documentation for fontspec explains how to redefine the hyphen character. It turns out that this seems to work similarly to the LaTeX trick. That is, it allows hyphenation in words which are themselves hyphenated (right at the end in this case, from TeX's point of view). I wasn't sure how to specify the alternative hyphen character correctly but, thanks to Khaled Hosny's comment, I think that it should be as follows:

\documentclass{scrartcl}
\usepackage[hmargin = 5cm]{geometry}
\usepackage{fontspec}

\begin{document}
\fontspec[Mapping=tex-text]{Latin Modern Roman}%
\addfontfeature{HyphenChar="2010}

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

---Hola, esto es un texto absurdo ---para ejemplificar lo que ocurreconestedocumento--- con algunas palabras más.

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
\end{document}

Here's the output:

This is not specific to Latin Modern. Just I couldn't figure out how to add a general font feature for all fonts. It seemed fontspec wanted me to specify a font to add the feature to. It should work for any font which includes U+2010. For example:

\documentclass{scrartcl}
\usepackage[hmargin = 5cm]{geometry}
\usepackage{fontspec}

\begin{document}
\fontspec[Mapping=tex-text]{Brill Roman}%
\addfontfeature{HyphenChar="2010}

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

---Hola, esto es un texto absurdo ---para ejemplificar lo que ocurreconestedocumento--- con algunas palabras más.

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
\end{document}

produces:

This solution is not as general as the LaTeX solution. The LaTeX solution can be made to work for any font which includes any hyphen character at all because you can use a single hyphen character twice when setting up the font for use with LaTeX. That is, you can just repeat the hyphen in slot 127 if the font doesn't have a second hyphen character itself. From TeX's point of view, the characters in slots 45 and 127 really are then different.

The solution I've given, in contrast, requires that the font actually have a second hyphen character in a suitable slot. (And the soft hyphen in U+00AD will not, it seems, work.) Nonetheless, it should work well for many fonts, especially fonts which are more likely to be used with TeX to typeset body text rather than, say, just a fancy heading where a font with very limited coverage might work. But in the case of a fancy heading, say, hyphenation is less likely to be a problem.

It would be nice to have a perfectly general solution but I'm not sure that is possible without re-engineering the core of TeX itself since, as I understand it, the prohibition on breaking already-hyphenated words is hard-coded and not alterable at the macro level. That is, you'd have to rewrite the relevant part of TeX's hyphenation algorithm to alter this.

EDIT: If you would like to type the emdash directly rather than typing ---, the following combines egreg's suggestion in the comments to JLDiaz's answer with the specification of hyphenchar suggested here:

\documentclass{scrartcl}
\usepackage[hmargin = 4cm]{geometry}
\usepackage{fontspec}
\usepackage{newunicodechar}

\newunicodechar—{{---}}

\begin{document}
\fontspec[Mapping=tex-text]{Latin Modern Roman}%
\addfontfeature{HyphenChar="2010}

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento— con algunas palabras más.

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento—, con algunas palabras más.

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
\end{document}

Output:

This also allows hyphenation where the emdash is directly followed by a comma, for example, as well as a line break where the emdash is directly followed by a space.

Answer 4

This is for XeLaTeX only, as the behavior of LuaLaTeX when inputting — (U+2014) seems satisfactory.

We want to allow hyphenation in the word preceding the em-dash, so we can add a zero kern before it, which will make the word end, but doesn't create a line break point.

However, we want also to remove the behavior of the em-dash that, for compatibility with classical TeX adds a discretionary. So we typeset the em-dash in a box.

Next we check whether the following token is a space; if it is, we do nothing, otherwise we add \nobreak\hspace{0pt}.

\documentclass{scrartcl}
\usepackage[hmargin = 4cm]{geometry}
\usepackage{amsmath}
\usepackage{fontspec}

\usepackage{newunicodechar}

\ExplSyntaxOn
\xetex_if_engine:T
 {
  \newunicodechar{—}{\fixed_dash:n { — }} % em-dash
  \newunicodechar{–}{\fixed_dash:n { – }} % en-dash
  \cs_new_protected:Npn \fixed_dash:n #1
   {
    \leavevmode\kern0pt~\mbox{ #1 }
    \peek_catcode:NF \c_space_token { \nobreak\hspace{0pt} }
   }
 }
\ExplSyntaxOff

\begin{document}

—Hola, esto es un texto absurdo —para ejemplificar lo que 
ocurreconestedocumento— con algunas palabras más.

—Hola, esto es un texto absurdo —para ejemplificar lo que 
—ocurreconestedocumento— con algunas palabras más.

—Hola, esto es un texto absurdo —para ejemplificar lo que 
—ocurrecon— con algunas palabras más.

—Hola, esto es un texto absurdo —para ejemplificar lo que 
—ocurrecon—, con algunas palabras más.

\end{document}

As you see, hyphenation is allowed also in the word following the em-dash, and the comma is kept in the same line. The output is identical with LuaLaTeX. No protrusion with microtype and XeLaTeX, of course.

Answer 5

It is of course not a general solution, and is in the spirit of some other suggestions, but a compensation of a space width gives proper hyphenation. The value depends on font.

\documentclass{scrartcl}
\usepackage[hmargin = 4cm]{geometry}
\usepackage{fontspec}

\begin{document}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

---Hola, esto es un texto absurdo ---para ejemplificar lo que ocurreconestedocumento \hspace{-0.33em}--- con algunas palabras más.

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
\end{document}

Answer 6

Concentrating on the direct use of emdash as control of --- is a bit harder.

The following produces the following in pdflatex, lualatex and xelatex. I think that's the desired outcome.

\documentclass{scrartcl}
\usepackage[hmargin = 4cm]{geometry}

\ifx\Umathchar\undefined
%pdftex
  \usepackage[utf8]{inputenc}
  \let\oldtextemdash\textemdash
  \def\textemdash{%
    \leavevmode\nobreak\hskip0pt\hbox{\oldtextemdash}%
    \nobreak\hskip0pt\special{}}
\else
  \usepackage{fontspec}
  \usepackage{newunicodechar}
  \ifx\directlua\undefined
%xetex
    \newunicodechar―{%
    \leavevmode\nobreak\hskip0pt\hbox{---}%
    \nobreak\hskip0pt\special{}}
  \else
% luatex
  \fi
\fi



    \begin{document}

    \subsection*{hyphenation in word before dash}

    ―Hola, esto es un texto absurdo ―para ejemplificar lo que
    ocurreconestedocumento― con algunas palabras más.


    \subsection*{break at space after dash}

    ―Hola,  absurdo ―para ejemplificar lo que
    aaaaa
    ocurreconestedocumento― con algunas palabras más.

    \subsection*{hyphenation in word after dash}

    ―Hola,  absurdo ―para ejemplificar lo que
    aa
    ocurreconestedocumento―cillum algunas palabras más.



    \end{document}