3

For a script, I want to get the hyphenation that LaTeX chooses, including my custom hyphenations file.

I use this code and it works mostly:

echo "\\\documentclass{standalone}
\\\usepackage[T1]{fontenc}
\\\usepackage[utf8]{inputenc}
\\\usepackage[ngerman]{babel}
\\\input{headers/hyphenation.tex}
\\\showhyphens{Nahrungsmittelunverträglichkeit}" | pdflatex | grep -a "T1/cmr/m/n/10" | sed -e 's/.*T1\/cmr\/m\/n\/10\s*//'

With most stuff, this works and only gives me the hyphenated version of that word. But when the word in \showhyphens contains things like umlauts, it replaces them with the UTF8 replacement character.

This does not occur if I do not pipe it into (pdf)LaTeX, so I guess the characters are properly submitted to LaTeX. But LaTeX cannot properly output them to the console.

How to get LaTeX to properly show this instead of

Nah-rungs-mit-te-lun-ver-tr�g-lich-keit

?

  • Welcome to TeX.SX! Unicode characters a bit complicated for TeX, have you tried one of the Unicode engines XeLaTeX or LuaLaTeX? – moewe Jun 21 '18 at 16:34
  • what OS are you using? – naphaneal Jun 21 '18 at 16:45
  • three backslashes in the echo? – David Carlisle Jun 21 '18 at 16:52
  • if I replace \\\ by \\ then your example gives an error that headers/hyphenation.tex is not found. please provide an example that may be run to debug the issue. – David Carlisle Jun 21 '18 at 16:53
  • If I change my terminal encoding to UTF-8 I at least can get XeLaTeX and LuaLaTeX to show the right thing in the output. (Again Windows 10, cmd terminal, MikTeX binaries.) – moewe Jun 21 '18 at 16:58
7

Hyphenation in TeX is (unfortunately) tied to the font encoding. You are using T1. \showoutput is showing the internal box representation of the characters so the fact that your input was utf-8 or even that it was ä is long gone. The internal T1 encoding that is being displayed has some resemblance to latin-1 but none at all to UTF-8, and this box logging is built in to the engine and not customisable by a macro layer such as latex.

LaTeX can output utf8 on the terminal via

\typeout{\detokenize{Nahrungsmittelunverträglichkeit}}

but that is outputting something completely different, the input character tokens, not an internal box representation.


If I change your script to

echo "\\documentclass{standalone}
\\usepackage[$1]{fontenc}
%\\usepackage[utf8]{inputenc}
\\usepackage[ngerman]{babel}
%\\input{headers/hyphenation.tex}
\\showhyphens{Nahrungsmittelunverträglichkeit}" | $2

Then

. test OT1 pdflatex

produces

[] \OT1/cmr/m/n/10 Nah-rungs-mit-te-lun-vertr^^?aglichkeit

and

. test T1 pdflatex

produces

[] \T1/cmr/m/n/10 Nah-rungs-mit-te-lun-ver-tr�g-lich-keit

and

. test TU xelatex

produces

\TU/lmr/m/n/10 Nah-rungs-mit-te-lun-ver-träg-lich-keit

So you see three different internal representations of the same input, the xetex one is essentially the unicode text so displays naturally.

  • what if you replace ä with \"a during read of input? – naphaneal Jun 21 '18 at 20:10
  • @naphaneal it makes no difference, inputenc expands ä to \"{a} so entering \"{a} just saves some expansion steps but the box representation shown by \showhyphens is not affected. – David Carlisle Jun 21 '18 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.