It doesn't hyphenate words ending with “—”

Question

UPDATE

Well. After a while, I decided to see again about this “problem”, and just discovered that in pdfLaTeX, using \usepackage[utf8]{inputenc} and the — (the unicode —, not the ligature ---) it works perfect (at least in what I tried). Minor edit: as @cfr mentions in her answer (I forgot it), it is in fact possible to use --- (the ligature) if you use T1 encoding and \hyphenchar\font=\string"7F. In both methods (wether I use the em-dash — or ---) microtype works perfectly.

Now, the problem remains in XeLaTeX. I would like to clear my idea: I want the em-dash to behave correctly; it doesn't matter if I have to type — (the unicode em-dash —) or --- (the usual LaTeX way); it should (among others) hyphenate words correctly; be always together to the word; in case it's followed by a comma/period, they should be together; and, microtype should work, e.g., the hyphen should still hang in the margin.

Of course, there is a basic solution (which works in any engine): indicate to XeLaTeX the breaking points, for instance ocur\-recone\-stedoc\-umento---. But I'm looking for an automatic solution.

If you have anything to say, please, say it, it's welcomed!

Note: Every — you see in the code is an em-dash.

I'writing a paper, and XeLaTeX doesn't hyphenate words which end with – (traditional LaTeX ---). After reading the comments, I realized this is a common problem also in LaTeX (not only XeLaTeX). Here it is a minimal working example:

This code (full example at the bottom of the question) outputs

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento— con algunas palabras más.

enter image description here

If we substitute the last — by a comma, for example it hyphenates correctly

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento, con algunas palabras más.

enter image description here

Moreover, if the phrase ends with —. XeLaTeX (or whoever is doing this) takes the full stop to the next line

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento—.

enter image description here

Here is a full minimal working example

\documentclass{scrartcl}
\usepackage[hmargin = 4cm]{geometry}
\usepackage{fontspec}

\begin{document}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento— con algunas palabras más.

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
\end{document}

Any ideas?

As a quick workaround try inserting an optional hyphen before the troublesome dash, to force the hand of the hyphenation routine; for example ocurreconeste\-documento---. — Thruston, Dec 16 '13 at 14:21
@morbusg Where should I put that? If I put it in the preamble, after the fontspec package (see the MWE), it does nothing. — Manuel, Dec 24 '13 at 15:58
Note that if you are talking latex or pdflatex, you can use --- if you use T1 and the alternate hyphenation character. That is, it isn't necessary to type the unicode emdash directly (although that will of course work). I just mention this as your update suggests that it isn't possible to avoid the problem if you stick to --- but that's not so. — cfr, May 3 '14 at 2:13
@cfr I forgot to say it! I will edit now. It's true, with pdfLaTeX (and LaTeX, as you say) you can type --- and get correct output. And, in pdfLaTeX with any of those solutions, microtype works correctly (I think), the punctuation hangs a little bit to the right :D Now the problem is XeLaTeX :P — Manuel, May 3 '14 at 10:15

wipet · Accepted Answer · 2015-04-19 06:29:42Z

6

You can prepare the following example:

\input ucode
\input lmfonts

\hsize=12cm

—Hola, esto es un texto absurdo —para ejemplificar lo que
ocurreconestedocumento— con algunas palabras más.

\end

and you can try to process it by 1) xetex test and 2) xetex -fmt pdfcsplain test. You will see different results: 1) the long word isn't hyphenated, 2) the long word is hyphenated.

You can study the difference in code settings, hyphenation settings etc. IMHO LuaLaTeX copies the settings from xetex's eplain, no from pdfcsplain. So, you have the problem.

Edit: Where is the difference? We can see the following setting in xetex.ini and in xelatex.ini:

 \XeTeXdashbreakstate=1

Bingo! Here is the problem. Set \XeTeXdashbreakstate=0 and your words will be hyphenated.

edited Apr 19 '15 at 6:29

answered Apr 19 '15 at 5:26

wipet

35.5k5083

I think you win, after more than a year of unanswered question. – Manuel Apr 19 '15 at 15:02
1

@Manuel I was not able to solve this problem before "more than a year" because I am monitoring this site less than one year:) – wipet Apr 19 '15 at 18:32
It was a general comment. A quite easy answer that did not appear until tons of months after the question (and three small bounties to encourage). – Manuel Apr 19 '15 at 20:56

add a comment |

JLDiaz · Accepted Answer · 2013-12-16 15:09:14Z

You can read an entry about this problem in my (now abandoned) blog, why it happens, and how it is solved in spanish babel.

But you are using xelatex and polyglossia, and I don't know if some solution is already included in this page. Anyway, it is easy to adapt the ideas and techniques used by babel, and define the following command:

\def\raya{%
\nobreak\hskip0pt\hbox{---}\nobreak\hskip0pt%
}

You have to put \raya{} instead of ---. So, in your example:

\raya{}Hola, esto es un texto absurdo \raya{}para ejemplificar lo que ocurreconestedocumento\raya{} con algunas palabras m
ás.

\raya{}Hola, esto es un texto absurdo \raya{}para ejemplificar lo que ocurreconestedocumento\raya{}, con algunas palabras 
más.

\raya{}Hola, esto es un texto absurdo \raya{}para ejemplificar lo que ocurreconestedocumento\raya{}.

And this is the output:

Result

Update

As requested by the OP,it is possible ~~to make unicode character — active (this is easy in xelatex, since it has native utf8 input)~~, to define — as a new Unicode char, and then use — instead of \raya [Thanks to egreg for pointing me to package newunicodechar, which is a cleaner solution than my previous attempt changing catcodes, and does not have issues with spaces after the character]:

\documentclass{scrartcl}
\usepackage[hmargin = 4cm]{geometry}
\usepackage{fontspec}
\usepackage{newunicodechar}

\newunicodechar—{%
\leavevmode\nobreak\hskip0pt\hbox{---}\nobreak\hskip0pt\relax%
}

\begin{document}
—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento— con algunas palabras más.

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento—, con algunas palabras más.

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento—.
\end{document}

Resulting in:

Result

\newunicodechar—{\leavevmode\nobreak\hskip0pt\hbox{---}\nobreak\hskip0pt\relax} (after loading newunicodechar). — egreg, Dec 16 '13 at 15:02
@egreg Great! Thanks for pointing to newunicodechar package. Updated answer. — JLDiaz, Dec 16 '13 at 15:09
@JLDiaz With \newunicodechar you just make the character active and define it as desired (actually \protected\def is used, so the active character won't expand in unsafe places); it's an easier wrapper, in my opinion. You can even say \hbox{—} (with an em-dash in the \hbox) instead of \hbox{---}. — egreg, Dec 16 '13 at 15:42
Unfortunately, this solution also prevents a line break after the dash if followed by a space. I wonder if the definition can be expanded with something similar to xpace. — Javier Bezos, Dec 16 '13 at 16:51
@Manuel Try the following, but it's mostly untested: \newunicodechar—{% \leavevmode\nobreak\hskip0pt\hbox{---}\nobreak\hskip0pt\relax \futurelet\esptestnext\esptestdo} \def\esptestdo{\ifcat\esptestnext\space\allowbreak\space\ignorespaces\fi} (incidentally, I don't know of any program handling correctly out of the box the em-dash according the Spanish rules). — Javier Bezos, Dec 29 '13 at 12:46

cfr · Accepted Answer · 2013-12-26 14:35:28Z

In regular TeX with the T1 encoding there is a way to avoid this problem which involves redefining the hyphen character. This is possible because T1 encodes two hyphen characters which look identical (in most fonts) but which play different roles.

The beauty of this is that it does not require the use of any special commands in the way the babel solution does. All you do is add a single line of code following \begin{document}:

\hyphenchar\font=\string"7F

after loading T1 with the fontenc package. What this does is tells TeX to use the character is slot 127 as the hyphenation character. That is, when TeX needs to break a word across lines, it will use "7F to hyphenate the word. It does not change the character you get when you type '-', however. That character corresponds to the one in slot 45 of the T1 encoding. So TeX does not see a word which is already hyphenated as hyphenated. Hence the prohibition on hyphenating already hyphenated words does not apply, and TeX breaks the word as appropriate. This also retains ligaturing since it is the character in slot 45 - not the one in slot 127 - which is defined in ligatures such as '--' and '---' in T1. So you can break the norms of English typesetting with impunity!

So I wondered if something similar might be possible with XeLaTeX as well. The documentation for fontspec explains how to redefine the hyphen character. It turns out that this seems to work similarly to the LaTeX trick. That is, it allows hyphenation in words which are themselves hyphenated (right at the end in this case, from TeX's point of view). I wasn't sure how to specify the alternative hyphen character correctly but, thanks to Khaled Hosny's comment, I think that it should be as follows:

\documentclass{scrartcl}
\usepackage[hmargin = 5cm]{geometry}
\usepackage{fontspec}

\begin{document}
\fontspec[Mapping=tex-text]{Latin Modern Roman}%
\addfontfeature{HyphenChar="2010}

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

---Hola, esto es un texto absurdo ---para ejemplificar lo que ocurreconestedocumento--- con algunas palabras más.

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
\end{document}

Here's the output:

enter image description here

This is not specific to Latin Modern. Just I couldn't figure out how to add a general font feature for all fonts. It seemed fontspec wanted me to specify a font to add the feature to. It should work for any font which includes U+2010. For example:

\documentclass{scrartcl}
\usepackage[hmargin = 5cm]{geometry}
\usepackage{fontspec}

\begin{document}
\fontspec[Mapping=tex-text]{Brill Roman}%
\addfontfeature{HyphenChar="2010}

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

---Hola, esto es un texto absurdo ---para ejemplificar lo que ocurreconestedocumento--- con algunas palabras más.

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
\end{document}

produces:

enter image description here

This solution is not as general as the LaTeX solution. The LaTeX solution can be made to work for any font which includes any hyphen character at all because you can use a single hyphen character twice when setting up the font for use with LaTeX. That is, you can just repeat the hyphen in slot 127 if the font doesn't have a second hyphen character itself. From TeX's point of view, the characters in slots 45 and 127 really are then different.

The solution I've given, in contrast, requires that the font actually have a second hyphen character in a suitable slot. (And the soft hyphen in U+00AD will not, it seems, work.) Nonetheless, it should work well for many fonts, especially fonts which are more likely to be used with TeX to typeset body text rather than, say, just a fancy heading where a font with very limited coverage might work. But in the case of a fancy heading, say, hyphenation is less likely to be a problem.

It would be nice to have a perfectly general solution but I'm not sure that is possible without re-engineering the core of TeX itself since, as I understand it, the prohibition on breaking already-hyphenated words is hard-coded and not alterable at the macro level. That is, you'd have to rewrite the relevant part of TeX's hyphenation algorithm to alter this.

EDIT: If you would like to type the emdash directly rather than typing ---, the following combines egreg's suggestion in the comments to JLDiaz's answer with the specification of hyphenchar suggested here:

\documentclass{scrartcl}
\usepackage[hmargin = 4cm]{geometry}
\usepackage{fontspec}
\usepackage{newunicodechar}

\newunicodechar—{{---}}

\begin{document}
\fontspec[Mapping=tex-text]{Latin Modern Roman}%
\addfontfeature{HyphenChar="2010}

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento— con algunas palabras más.

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento—, con algunas palabras más.

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
\end{document}

Output:

enter image description here

This also allows hyphenation where the emdash is directly followed by a comma, for example, as well as a line break where the emdash is directly followed by a space.

What does it do? How does it work. It seems to work right BUT it's not available —the alternative hyphen char— in many fonts. If you could explain what exactly happens, then may be there's a hack for those fonts which are missing that symbol. — Manuel, Dec 23 '13 at 0:47
OK. I don't know exactly what it is doing because I don't understand XeLaTeX well enough. Here is the reference which gave me the idea tex.stackexchange.com/questions/63232/…, combined with the fontspec documentation. I think most fonts define a second hyphen of some kind which is essentially just like the first one e.g. non-breaking hyphen etc. I'll see if I can think of anything better, though. Is there a way using fontspec to replicate a character to an empty slot? That would allow a perfectly general solution as with T1. — cfr, Dec 23 '13 at 1:52
tug.org/mailman/htdig/xetex/2006-November/005435.html is interesting, too. — cfr, Dec 25 '13 at 4:21
Well, after some testing, the last example you provided works perfectly (same as the solution of Javier Bezos) BUT if I use microtype the new HyphenChar doesn't go into the margin… If you solve that, I don't know what answer should I accept. Thank you so much. — Manuel, Dec 29 '13 at 15:52

egreg · Accepted Answer · 2015-04-18 22:49:46Z

This is for XeLaTeX only, as the behavior of LuaLaTeX when inputting — (U+2014) seems satisfactory.

We want to allow hyphenation in the word preceding the em-dash, so we can add a zero kern before it, which will make the word end, but doesn't create a line break point.

However, we want also to remove the behavior of the em-dash that, for compatibility with classical TeX adds a discretionary. So we typeset the em-dash in a box.

Next we check whether the following token is a space; if it is, we do nothing, otherwise we add \nobreak\hspace{0pt}.

\documentclass{scrartcl}
\usepackage[hmargin = 4cm]{geometry}
\usepackage{amsmath}
\usepackage{fontspec}

\usepackage{newunicodechar}

\ExplSyntaxOn
\xetex_if_engine:T
 {
  \newunicodechar{—}{\fixed_dash:n { — }} % em-dash
  \newunicodechar{–}{\fixed_dash:n { – }} % en-dash
  \cs_new_protected:Npn \fixed_dash:n #1
   {
    \leavevmode\kern0pt~\mbox{ #1 }
    \peek_catcode:NF \c_space_token { \nobreak\hspace{0pt} }
   }
 }
\ExplSyntaxOff

\begin{document}

—Hola, esto es un texto absurdo —para ejemplificar lo que 
ocurreconestedocumento— con algunas palabras más.

—Hola, esto es un texto absurdo —para ejemplificar lo que 
—ocurreconestedocumento— con algunas palabras más.

—Hola, esto es un texto absurdo —para ejemplificar lo que 
—ocurrecon— con algunas palabras más.

—Hola, esto es un texto absurdo —para ejemplificar lo que 
—ocurrecon—, con algunas palabras más.

\end{document}

enter image description here

As you see, hyphenation is allowed also in the word following the em-dash, and the comma is kept in the same line. The output is identical with LuaLaTeX. No protrusion with microtype and XeLaTeX, of course.

If no one finds a problem here, I will accept it. I wish I could remove all comments from this question (which is rather messed up). It's a pity that no microtype-compatible solution exists. By the way, could you add an explanation of why \kern0pt (because JLDiaz had \nobreak\hskip0pt, and that seemed not to give any problems). The last \hspace is because \nobreak only acts on skips, so we need to put one there (of zero width)? — Manuel, Apr 19 '15 at 0:47
A kern is never a line break point, unless it is followed by glue, so you get hyphenation in the preceding word, but the last fragment is not separated by the dash. I added the test for a space; it's not complete, because only an explicit space token allows a line break after the dash (not \quad, for instance). This test is necessary for adding a \nobreak in case a character directly follows. The fact that even —\nobreak, can't inhibit the break is a problem also in standard TeX (with ---,). — egreg, Apr 19 '15 at 9:23

Przemysław Scherwentke · Accepted Answer · 2013-12-25 06:48:09Z

It is of course not a general solution, and is in the spirit of some other suggestions, but a compensation of a space width gives proper hyphenation. The value depends on font.

\documentclass{scrartcl}
\usepackage[hmargin = 4cm]{geometry}
\usepackage{fontspec}

\begin{document}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

---Hola, esto es un texto absurdo ---para ejemplificar lo que ocurreconestedocumento \hspace{-0.33em}--- con algunas palabras más.

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
\end{document}

At first sight I thought this was something manual that I wouldn't like. But may be with some working this could work. Is there a way we could add not -0.33em but the exact width of an space (\widthof{~} or something else)? In that case, may be mixing this solution with the @JLDiaz would work: \newunicodechar—{~\removeonespace—}. That wouldn't touch the hyphenation and could work… I would appreciate a little deeper explanation. — Manuel, Dec 26 '13 at 16:25
@Manuel In plain TeX \spaceskip is defined as 0.3333em. In ordinary fonts it should be easy to establish the relation between space and em. For advanced fonts I am not sure, alas. And certainly in my suggestion there would be finally a definition replacing ---, but the answer only suggest an idea of changing. — Przemysław Scherwentke, Dec 27 '13 at 2:11
But the spaces are someting like 0.3333em plus X minus Y so it stretches reasonably. That's why I asked. — Manuel, Dec 27 '13 at 11:27
\hskip-\lastskip but the method here doesn't remove the breakpoint before the dash — David Carlisle, Apr 19 '15 at 16:45

David Carlisle · Accepted Answer · 2015-04-19 09:16:02Z

Concentrating on the direct use of emdash as control of --- is a bit harder.

The following produces the following in pdflatex, lualatex and xelatex. I think that's the desired outcome.

enter image description here

\documentclass{scrartcl}
\usepackage[hmargin = 4cm]{geometry}

\ifx\Umathchar\undefined
%pdftex
  \usepackage[utf8]{inputenc}
  \let\oldtextemdash\textemdash
  \def\textemdash{%
    \leavevmode\nobreak\hskip0pt\hbox{\oldtextemdash}%
    \nobreak\hskip0pt\special{}}
\else
  \usepackage{fontspec}
  \usepackage{newunicodechar}
  \ifx\directlua\undefined
%xetex
    \newunicodechar―{%
    \leavevmode\nobreak\hskip0pt\hbox{---}%
    \nobreak\hskip0pt\special{}}
  \else
% luatex
  \fi
\fi



    \begin{document}

    \subsection*{hyphenation in word before dash}

    ―Hola, esto es un texto absurdo ―para ejemplificar lo que
    ocurreconestedocumento― con algunas palabras más.


    \subsection*{break at space after dash}

    ―Hola,  absurdo ―para ejemplificar lo que
    aaaaa
    ocurreconestedocumento― con algunas palabras más.

    \subsection*{hyphenation in word after dash}

    ―Hola,  absurdo ―para ejemplificar lo que
    aa
    ocurreconestedocumento―cillum algunas palabras más.



    \end{document}

asked	5 years, 5 months ago
viewed	1,011 times
active	4 years, 1 month ago

Stack Exchange Network

current community

your communities

more stack exchange communities

It doesn't hyphenate words ending with “—”

UPDATE

Any ideas?

6 Answers 6

Update

Your Answer

Not the answer you're looking for? Browse other questions tagged spacing xetex fontspec hyphenation spanish or ask your own question.

Linked

Hot Network Questions

It doesn't hyphenate words ending with “—”

UPDATE

Any ideas?

6 Answers 6

Update

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged spacing xetex fontspec hyphenation spanish or ask your own question.

Linked

Related

Hot Network Questions