How to convert all hyphenation patterns from Unicode to T1 in lualatex?

Question

I'm trying to use lualatex with old, 8-bit, non-Unicode fonts:

\documentclass[a4paper]{article}
\usepackage[utf8]{luainputenc}
\usepackage{t1enc}
\usepackage{lmodern}
\def\myhelvetica{\fontfamily{phv}\selectfont}
\begin{document}
\makeatletter\language=\l@hungarian
\showhyphens{árvíztűrő türkörfúrógép}
\hrule
{\Huge Helló, Wörld, árvíztűrő}
\hrule
{\Huge Helló, Wörld, árvízt\H{u}r\H{o}}
\hrule
{\myhelvetica\Huge Helló, Wörld, árvíztűrő}
\hrule
\end{document}

Everything works except that \showhyphens shows incorrect hyphenation:

• ár-víz-tű-rő would be correct, \showhyphens shows something else. (When using lualatex and fontspec, this is also hyphenated correctly.) I want \showhypens to show ár-víz-tű-rő here in lualatex, even without fontspec.
• tü-kör-fú-ró-gép is hyphenated correctly.

I suspect that the problem is caused by the letters ő and ű, which are at different positions in Unicode and T1 encodings. (All other letters, including ó and é are at the same position.)

Is there a way to convert all hyphenation patterns from Unicode to T1 encoding, something I can add between \documentclass and \begin{document}? I won't be using fontspec or any Unicode font in this compilation. The files containing the hyphenation patterns are probably correct, I just want lualatex to load them with a different encoding (T1), or do an in-memory encoding conversion after having loaded them. It's OK for me if some extra Lua code has to be written to make this happen.

I know and I've checked that it works in pdflatex out of the books. However, in this question I'm looking for a way to make it work in lualatex. I'm not looking for advice whether I should be using pdflatex or lualatex: for the purposes of this question I'm committed to lualatex.

Answer 1

I was able to solve this in the meantime. Here is the solution (with important lines indicated with a trailing %%):

% pl.tex, works with modern pdflatex and lualatex in TeX Live 2019.
\documentclass{article}
\usepackage[utf8]{luainputenc}  %%
\usepackage{lmodern} \normalfont
\usepackage{t1enc}
\let\kanjiskip\hsize  %%
\usepackage[hungarian]{babel}
\expandafter\let\expandafter\CCC\csname @tempcnta\endcsname  %%
\CCC128 \loop\ifnum\CCC<159 \ifnum\lccode\CCC=0 \lccode\CCC\CCC\fi  %%
  \advance\CCC1 \repeat  %%
\CCC160 \loop\ifnum\CCC<190 \ifnum\lccode\CCC=0 \lccode\CCC\CCC\fi  %% 
  \advance\CCC1 \repeat  %%
\CCC192 \loop\ifnum\CCC<256 \ifnum\lccode\CCC=0 \lccode\CCC\CCC\fi  %%
  \advance\CCC1 \repeat  %%
\begin{document}
\showhyphens{árvíztűrő tükörfúrógép}
\hrule width \hsize
\bigskip
\vbox{\Huge\hsize7em\hrule\noindent . . . . . bababababababa\hrule}
\bigskip
\vbox{\Huge\hsize7em\hrule\noindent . . . . . bábábábábábábá\hrule}
\bigskip
\vbox{\Huge\hsize7em\hrule\noindent . . . . . bőbőbőbőbőbőbő\hrule}
\bigskip
\vfill\hrule width\hsize
\end{document}

It hyphenates as expected: ba-ba-..., bá-bá-..., bő-bő-..., ár-víz-tű-rő tü-kör-fú-ró-gép.

FYI here is my original example, which works with pdflatex only:

% p.tex, works with pdflatex only.
\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{lmodern} \normalfont
\usepackage{t1enc}
\usepackage[hungarian]{babel}
\begin{document}
\showhyphens{árvíztűrő tükörfúrógép}
\hrule width \hsize
\bigskip
\vbox{\Huge\hsize7em\hrule\noindent . . . . . bababababababa\hrule}
\bigskip
\vbox{\Huge\hsize7em\hrule\noindent . . . . . bábábábábábábá\hrule}
\bigskip
\vbox{\Huge\hsize7em\hrule\noindent . . . . . bőbőbőbőbőbőbő\hrule}
\bigskip
\vfill\hrule width\hsize
\end{document}

Explanation of the solution (pl.tex):

• Input files are in UTF-8 encoding, text fonts are in T1 encoding.
• \usepackage[utf8]{luainputenc}: luainputenc.sty is needed instead of inputenc.sty, otherwise with lualatex the letter ő and ű wouldn't appear at all.
• \let\kanjiskip\hsize: This makes hyph-hu.ec.tex be loaded instead of hyph-hu.tex by loadhyph-hu.tex, loaded indirectly by babel.sty. We need hyphu-hu.ec.tex, because it contains the hyphenation patterns in the T1 encoding (rather than UTF-8).
• \lccode\CCC\CCC, e.g. \lccode"AE="AE: Without this lualatex wouldn't recognize ő and ű as letters when hyphenating. AE is the hex code of ő in T1 encoding.
• In the console log output of pdflatex in an UTF-8 terminal window, the \showhyphens line contains all accented characters incorrectly: �r-v�z-t�-r� t�-k�r-f�-r�-g�p . Nevertheless, the words are correctly hyphenated.
• In the console log output of lualatex in an UTF-8 terminal window, the \showhyphens line contains the some (ő and ű) accented characters incorrectly: ár-víz-t¶-r® tü-kör-fú-ró-gép. Nevertheless, the words are correctly hyphenated. The reason why ő and ű are incorrect is because the console log output shows characters in T1 encoding (mapped to UTF-8), and ő and ű are the characters which have a different position in T1 vs Unicode.

Answer 2

This doesn't answer the question directly, but it shows some insights on how I debugged \showhyphens.

I was using this source file:

% Compile it with lualatex.
\documentclass[a4paper]{article}
\usepackage[utf8]{luainputenc}
\usepackage{t1enc}
\usepackage{lmodern}
\usepackage[hungarian,english]{babel}
\begin{document}
\selectlanguage{hungarian}
% Correct hyphenation is ár-víz-tű-rő tü-kör-fú-ró-gép
\showhyphens{árvíztűrő tükörfúrógép}  % Incorrect.
\showhyphens{árvízt\string űr\string ő türkörfúrógép}  % Correct.
{\lccode\string`X="0171 \lccode\string`Y="0151
\lowercase{\showhyphens{árvíztXrY türkörfúrógép}}}  % Correct.
\edef\zzzz{ő}
\typeout{D1: \meaning ő}
\typeout{D2: \number\string`ő}
\typeout{D3: \meaning\zzzz}
\typeout{D4: \expandafter\number\expandafter\string\expandafter`\zzzz}
\hrule
{\Huge árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő
árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő
árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő
árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő
árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő
árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő
árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő árvíztűrő}
\hrule
\end{document}

The console log output:

D1: macro:->\H o
D2: 337
D3: macro:->®
D4: 174

My explanation:

• D2 shows that ő has character code 337, which corresponds to U+0151, the letter ő in Unicode.
• D1 shows that ő is an active character which expands to \H o.
• D3 shows that the full expansion of ő (by expanding \H o as much as possible) is ®.
• D4 shows that the character code of ® is 174, which corresponds to the letter ő in the T1 encoding (https://en.wikipedia.org/wiki/Cork_encoding).
• This is in line with what I've suspected in the question: I suspect that the problem is caused by the letters ő and ű, which are at different positions in Unicode and T1 encodings. (All other letters, including ó and é are at the same position.)
• When the argument of \showhyphens is expanded, ő is transformed to ®, and we get incorrect hyphenation. Hyphenation patterns are in Unicode in lualatex.
• When the argument of \showhyphens is expanded, \string ő is transformed to ő, and we get correct hyphenation.
• Similarly, with the \lccode trick above we can get a non-expanding ő (U+0151, Unicode 337) token, and we get correct hyphenation.
• The effect of \selectlanguage{hungarian} is different from the effect of \makeatletter\language=\l@hungarian, and the first one is correct. The second one prevents the tű-rő hyphenation.
• Even if we do \usepackage[utf8]{luainputenc} and \usepackage{lmodern} before loading babel, hyphenation patterns will be in Unicode.
• The file hyph-hu.tex contains Hungarian hyphenation patterns in Unicode (UTF-8), loadhyph-hu.tex takes care of loading it without encoding changes. Please note that there are other input files (e.g. hyph-hu.ec.tex) and there are encoding conversions, but they are not used.

Some more info on how lualatex loads the hyphenation patterns:

\makeatletter
\def\bbl@trace#1{}
\input luababel.def  % Loads language.dat and defines \l@hungarian.
\language=\l@hungarian
\input loadhyph-hu.tex  % Defines \patterns{...} in Unicode (UTF-8).
\lefthyphenmin=1
\righthyphenmin=1
\showhyphens{árvíztűrő türkörfúrógép}  % Correct.
\@@end

Normally we wouldn't have to load luababel.def and loadhyph-hu.tex manually, \usepackage[hungarian]{babel} would take care of that.