# How to convert all hyphenation patterns from Unicode to T1 in lualatex?

I'm trying to use lualatex with old, 8-bit, non-Unicode fonts:

\documentclass[a4paper]{article}
\usepackage[utf8]{luainputenc}
\usepackage{t1enc}
\usepackage{lmodern}
\def\myhelvetica{\fontfamily{phv}\selectfont}
\begin{document}
\makeatletter\language=\l@hungarian
\showhyphens{árvíztűrő türkörfúrógép}
\hrule
{\Huge Helló, Wörld, árvíztűrő}
\hrule
{\Huge Helló, Wörld, árvízt\H{u}r\H{o}}
\hrule
{\myhelvetica\Huge Helló, Wörld, árvíztűrő}
\hrule
\end{document}


Everything works except that \showhyphens shows incorrect hyphenation:

• ár-víz-tű-rő would be correct, \showhyphens shows something else. (When using lualatex and fontspec, this is also hyphenated correctly.) I want \showhypens to show ár-víz-tű-rő here in lualatex, even without fontspec.
• tü-kör-fú-ró-gép is hyphenated correctly.

I suspect that the problem is caused by the letters ő and ű, which are at different positions in Unicode and T1 encodings. (All other letters, including ó and é are at the same position.)

Is there a way to convert all hyphenation patterns from Unicode to T1 encoding, something I can add between \documentclass and \begin{document}? I won't be using fontspec or any Unicode font in this compilation. The files containing the hyphenation patterns are probably correct, I just want lualatex to load them with a different encoding (T1), or do an in-memory encoding conversion after having loaded them. It's OK for me if some extra Lua code has to be written to make this happen.

I know and I've checked that it works in pdflatex out of the books. However, in this question I'm looking for a way to make it work in lualatex. I'm not looking for advice whether I should be using pdflatex or lualatex: for the purposes of this question I'm committed to lualatex.

• From my point of view, nothing's wrong. – user156344 Jan 11 '19 at 11:10
• Why do you want to use lualatex when not using fontspec? In such qa case use simply pdflatex – user2478 Jan 11 '19 at 11:15
• Ok, but using lualatex with the old stuff luainputenc and t1enc makes no sense! – user2478 Jan 11 '19 at 11:22
• I really would not do this at all, but you don't need to convert anything all the hyphenation patterns are already available for T1 just input the file you would use for pdftex (in luatex you can do this at run time) especially if you are really using latin modern which is available as opentype there is no reason to do this at all – David Carlisle Jan 11 '19 at 11:58
• Glad you got something working, but I would still say that the correct answer would be not to use luainputenc and T1 encoding which between them disable almost all of luatex's font handling. – David Carlisle Jan 13 '19 at 16:49

I was able to solve this in the meantime. Here is the solution (with important lines indicated with a trailing %%):

% pl.tex, works with modern pdflatex and lualatex in TeX Live 2019.
\documentclass{article}
\usepackage[utf8]{luainputenc}  %%
\usepackage{lmodern} \normalfont
\usepackage{t1enc}
\let\kanjiskip\hsize  %%
\usepackage[hungarian]{babel}
\expandafter\let\expandafter\CCC\csname @tempcnta\endcsname  %%
\CCC128 \loop\ifnum\CCC<159 \ifnum\lccode\CCC=0 \lccode\CCC\CCC\fi  %%
\CCC160 \loop\ifnum\CCC<190 \ifnum\lccode\CCC=0 \lccode\CCC\CCC\fi  %%
\CCC192 \loop\ifnum\CCC<256 \ifnum\lccode\CCC=0 \lccode\CCC\CCC\fi  %%
\begin{document}
\showhyphens{árvíztűrő tükörfúrógép}
\hrule width \hsize
\bigskip
\vbox{\Huge\hsize7em\hrule\noindent . . . . . bababababababa\hrule}
\bigskip
\vbox{\Huge\hsize7em\hrule\noindent . . . . . bábábábábábábá\hrule}
\bigskip
\vbox{\Huge\hsize7em\hrule\noindent . . . . . bőbőbőbőbőbőbő\hrule}
\bigskip
\vfill\hrule width\hsize
\end{document}


It hyphenates as expected: ba-ba-..., bá-bá-..., bő-bő-..., ár-víz-tű-rő tü-kör-fú-ró-gép.

FYI here is my original example, which works with pdflatex only:

% p.tex, works with pdflatex only.
\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{lmodern} \normalfont
\usepackage{t1enc}
\usepackage[hungarian]{babel}
\begin{document}
\showhyphens{árvíztűrő tükörfúrógép}
\hrule width \hsize
\bigskip
\vbox{\Huge\hsize7em\hrule\noindent . . . . . bababababababa\hrule}
\bigskip
\vbox{\Huge\hsize7em\hrule\noindent . . . . . bábábábábábábá\hrule}
\bigskip
\vbox{\Huge\hsize7em\hrule\noindent . . . . . bőbőbőbőbőbőbő\hrule}
\bigskip
\vfill\hrule width\hsize
\end{document}


Explanation of the solution (pl.tex):

• Input files are in UTF-8 encoding, text fonts are in T1 encoding.
• \usepackage[utf8]{luainputenc}: luainputenc.sty is needed instead of inputenc.sty, otherwise with lualatex the letter ő and ű wouldn't appear at all.
• \let\kanjiskip\hsize: This makes hyph-hu.ec.tex be loaded instead of hyph-hu.tex by loadhyph-hu.tex, loaded indirectly by babel.sty. We need hyphu-hu.ec.tex, because it contains the hyphenation patterns in the T1 encoding (rather than UTF-8).
• \lccode\CCC\CCC, e.g. \lccode"AE="AE: Without this lualatex wouldn't recognize ő and ű as letters when hyphenating. AE is the hex code of ő in T1 encoding.
• In the console log output of pdflatex in an UTF-8 terminal window, the \showhyphens line contains all accented characters incorrectly: �r-v�z-t�-r� t�-k�r-f�-r�-g�p . Nevertheless, the words are correctly hyphenated.
• In the console log output of lualatex in an UTF-8 terminal window, the \showhyphens line contains the some (ő and ű) accented characters incorrectly: ár-víz-t¶-r® tü-kör-fú-ró-gép. Nevertheless, the words are correctly hyphenated. The reason why ő and ű are incorrect is because the console log output shows characters in T1 encoding (mapped to UTF-8), and ő and ű are the characters which have a different position in T1 vs Unicode.
• Have you tried \showhyphens{\detokenize{árvíztűrő tükörfúrógép}}, I can't test on TL2018 because I did not install all language support. Works on TL2016 but this is old lualatex. – user4686 Jan 13 '19 at 13:30
• @jfbu: Thanks for tip! I'll try this and some others, and report back with my findings. – pts Jan 13 '19 at 13:32

This doesn't answer the question directly, but it shows some insights on how I debugged \showhyphens.

I was using this source file:

% Compile it with lualatex.
\documentclass[a4paper]{article}
\usepackage[utf8]{luainputenc}
\usepackage{t1enc}
\usepackage{lmodern}
\usepackage[hungarian,english]{babel}
\begin{document}
\selectlanguage{hungarian}
% Correct hyphenation is ár-víz-tű-rő tü-kör-fú-ró-gép
\showhyphens{árvíztűrő tükörfúrógép}  % Incorrect.
\showhyphens{árvízt\string űr\string ő türkörfúrógép}  % Correct.
{\lccode\stringX="0171 \lccode\stringY="0151
\lowercase{\showhyphens{árvíztXrY türkörfúrógép}}}  % Correct.
\edef\zzzz{ő}
\typeout{D1: \meaning ő}
\typeout{D2: \number\stringő}
\typeout{D3: \meaning\zzzz}
\typeout{D4: \expandafter\number\expandafter\string\expandafter\zzzz}
\hrule
{\Huge árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő
árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő
árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő
árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő
árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő
árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő
árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő}
\hrule
\end{document}


The console log output:

D1: macro:->\H o
D2: 337
D3: macro:->®
D4: 174


My explanation:

• D2 shows that ő has character code 337, which corresponds to U+0151, the letter ő in Unicode.
• D1 shows that ő is an active character which expands to \H o.
• D3 shows that the full expansion of ő (by expanding \H o as much as possible) is ®.
• D4 shows that the character code of ® is 174, which corresponds to the letter ő in the T1 encoding (https://en.wikipedia.org/wiki/Cork_encoding).
• This is in line with what I've suspected in the question: I suspect that the problem is caused by the letters ő and ű, which are at different positions in Unicode and T1 encodings. (All other letters, including ó and é are at the same position.)
• When the argument of \showhyphens is expanded, ő is transformed to ®, and we get incorrect hyphenation. Hyphenation patterns are in Unicode in lualatex.
• When the argument of \showhyphens is expanded, \string ő is transformed to ő, and we get correct hyphenation.
• Similarly, with the \lccode trick above we can get a non-expanding ő (U+0151, Unicode 337) token, and we get correct hyphenation.
• The effect of \selectlanguage{hungarian} is different from the effect of \makeatletter\language=\l@hungarian, and the first one is correct. The second one prevents the tű-rő hyphenation.
• Even if we do \usepackage[utf8]{luainputenc} and \usepackage{lmodern} before loading babel, hyphenation patterns will be in Unicode.
• The file hyph-hu.tex contains Hungarian hyphenation patterns in Unicode (UTF-8), loadhyph-hu.tex takes care of loading it without encoding changes. Please note that there are other input files (e.g. hyph-hu.ec.tex) and there are encoding conversions, but they are not used.

\makeatletter

Normally we wouldn't have to load luababel.def and loadhyph-hu.tex manually, \usepackage[hungarian]{babel} would take care of that.