# Why does inputenc abandon so quickly under “utf8 based engines”?

Why do I need to do the extra work starting with \ifdefined in order to get my French guillemets correct in the pdf output, when using xelatex with a source specifying the use of T1-encoded fonts ?

\documentclass[french]{article}

\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}

\ifdefined\XeTeXinterchartoks
\catcode« \active
\catcode» \active
\def«{\char19 }
\def»{\char20 }% ça marche, même avec Babel+frenchb
\fi

\usepackage{newtxtext}

\usepackage{babel}
\frenchbsetup{og=«, fg=»}

\begin{document}

\showboxdepth\maxdimen
\showoutput

«coucou»
\end{document}


The log contains:

Package: inputenc 2015/03/17 v1.2c Input encoding file
\inpenc@prehook=\toks14
\inpenc@posthook=\toks15

Package inputenc Warning: inputenc package ignored with utf8 based engines.


But it is loaded after fontenc. It is not forbidden to use fontenc with xelatex. inputenc is loaded after it. Thus it should know that T1-encoded font slots are to be used. Why then doesn't it do the job of making these characters active and map them to the suitable \char xx slots ?

There is something escaping me here...

Notice that the code sample also uses babel+frenchb which adds automatic spacing. It seems not to have been perturbed from my making the characters active.

In order to explain more the issue, consider the following input:

\documentclass{article}

\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}

\begin{document}

\showboxdepth\maxdimen
\showoutput

«coucou»

\end{document}


It produces, if compiled with xelatex:

The explanation is simple: the ascii chars « and » are in slots 171 and 187 respectively. Hence the corresponding glyphs from the T1 encoding are used, giving the result. inputenc does nothing, but it could have donc something akin to my code above.

...\hbox(6.63332+0.0)x345.0, glue set 290.00977fil
....\hbox(0.0+0.0)x15.0
....\T1/cmr/m/n/10 «
....\T1/cmr/m/n/10 c
....\T1/cmr/m/n/10 o
....\T1/cmr/m/n/10 u
....\T1/cmr/m/n/10 c
....\T1/cmr/m/n/10 o
....\T1/cmr/m/n/10 u
....\T1/cmr/m/n/10 »

• Note that if you use default texlive settings, hyphenation patterns for T1 are not loaded into xetex or xelatex so hyphenation will be incorrect with T1 fonts. why use newtxtext with xetex? rather than Times ? – David Carlisle Dec 11 '15 at 18:30
• @DavidCarlisle No specific reason except from where the situation came from. Good to know about the missing hyphenation patterns, didn't know about that. Does this mean then that xelatex should only be used with Unicode fonts ? – jfbu Dec 11 '15 at 18:33
• @jfbu simple answer yes, longer answer it would be possible to load T1 hypenation patterns but it complicates things (and there is no "out of the box" setup for that) and doing it is very low priority as almost all relevant fonts are available as opentype fonts by now. – David Carlisle Dec 11 '15 at 18:34
• I think you are looking for the xetex equivalent of luainputenc. The xetex-inputenc package referenced there unfortunately only does anything useful with 8-bit input. – Robert Dec 11 '15 at 20:17
• PS. 7 years ago, there was a lengthy discussion about inputenc's future with UTF-aware engines here. – Robert Dec 11 '15 at 20:52

inputenc is abandoned because it does absolutely nothing with XeTeX or LuaLaTeX. Better said, it would do bad!

Essentially, the task performed by inputenc is translating input characters into their LICR form. With an 8 bit engine, « is two byte long and inputenc is able to translate them into \guillemotleft and » into \guillemotright. But for doing so it must make some characters active. Which is exactly what you do later on, and inputenc is not instructed to do, because it's thought for an 8 bit engine.

I added a friendlier interface with newunicodechar.

\documentclass[french]{article}

\usepackage[T1]{fontenc}
\usepackage{newunicodechar}

\newunicodechar{«}{\guillemotleft}
\newunicodechar{»}{\guillemotright}

\usepackage{newtxtext}

\usepackage{babel}
\frenchbsetup{og=«, fg=»}

\begin{document}

«coucou»

\end{document}


If your aim is to provide translations for the characters in t1enc.dfu, then you can use it in a different way.

\documentclass[french]{article}

\usepackage[T1]{fontenc}
\usepackage{newunicodechar}

\newcommand\DeclareUnicodeCharacter[2]{%
\expandafter\newunicodechar\Uchar"#1{#2}%
}
\input{t1enc.dfu}

\usepackage{newtxtext}

\usepackage{babel}
\frenchbsetup{og=«, fg=»}

\begin{document}

«coucou»

\end{document}


## A proof of concept for a package xeinputenc

\ProvidesPackage{xeinputenc}[2015/12/12]
\RequirePackage{newunicodechar}

\AtBeginDocument{\xeinputenc@process}

\newcommand{\xeinputenc@process}{%
\begingroup
\gdef\xeinputenc@list{}%
\def\cdp@elt##1##2##3##4{%
}%
\cdp@list
\aftergroup\xeinputenc@list
\endgroup
}

\newcommand{\DeclareUnicodeCharacter}[2]{%
\expandafter\newunicodechar\Uchar"#1{#2}%
}

\newcommand{\xeinputenc@input}[1]{%
\InputIfFileExists
{#1enc.dfu}
{\wlog{... processing UTF-8 mapping file for font encoding #1}\catcode\ 9\relax}%
{\wlog{... no UTF-8 mapping file for font encoding #1}}%
}

\@onlypreamble\DeclareUnicodeCharacter
\@onlypreamble\xeinputenc@list
\@onlypreamble\xeinputenc@process
\@onlypreamble\xeinputenc@input
\endinput


Now your test document can be

\documentclass[french]{article}

\usepackage{xeinputenc}

\usepackage{newtxtext}

\usepackage{babel}
\frenchbsetup{og=«, fg=»}

\begin{document}

«coucou»

\end{document}


No explicit loading of fontenc is needed in this case, because this is already taken care of by newtxtext, but calls to it will be honored.

• I am asking why inputenc does not do anything useful when detecting the user choice of T1 encoded fonts (a detection that it does). This is my question "why does it abandon so quickly". Someone has to do the job, and inputenc rather than doing nothing at all, could do something. I am not asking inputenc to insist doing the things it does under pdflatex: I am asking why does it not do something rather than nothing. There is an opportunity for inputenc to do something useful, if one follows the recommendation of loading it after fontenc. – jfbu Dec 11 '15 at 18:39
• @jfbu: I now see what you mean. Well it wouldn't a good idea if inputenc would make a lot chars active only because it sees T1-encoding. Your wishes are rather special (and not really recommendable) and not something for a standard package. – Ulrike Fischer Dec 11 '15 at 18:45
• @UlrikeFischer yes, usually about 128 or 128+32 characters could be made active to use the appropriate slots in the 256 slots font, alongside the core 7bit ascii chars. This is what I am asking. – jfbu Dec 11 '15 at 18:49
• @jfbu inputenc is not designed to do that! – egreg Dec 11 '15 at 18:51
• yes, but this is exactly what i asked ? "why does inputenc so stubbornly stick to its original design, which in the case at hand had the only logical conclusion to do nothing at all!! " but the situation being so much different, the raison d'être of the initial design from other contexts having vanished, isn't it time to re-invent something... that was the whole point of my question, and after much resistance I can now applaud that you do provide a method, less naïve than the one in my OP. Thanks ! – jfbu Dec 11 '15 at 19:03

inputenc's utf8 option is designed to take sequences of characters representing the bytes in utf8 representation as individual characters and collect them together and use the utf8 encoding to expand each such sequence into a suitable tex command for that character.

When a utf8 file is read by xetex, each character is reported as a single character token and the bytes in the utf8 encoding are not reported at all to the macro layer so the inputenc code can do nothing useful.

• inputenc normally takes into account the loaded font encodings. My question is: why does its utf8 option only target Unicode encoding ? that it does so is a choice. – jfbu Dec 11 '15 at 18:29
• @jfbu no, inputenc is independent of font encodings. – David Carlisle Dec 11 '15 at 18:31
• @jfbu utf8 option decodes utf8 encoded input. that is a sort of a choice but what else do you expect it to do? – David Carlisle Dec 11 '15 at 18:32
• As I said, inputenc does look at the loaded font encodings. As there is nothing to do under xelatex, the Unicode being treated natively, I would expect inputenc to do the appropriate work compatible with this user choice of using T1 encoded fonts. The reason being that nobody else is doing that for the poor user ! – jfbu Dec 11 '15 at 18:35
• @jfbu you know I wrote parts of this code:-) inputenc.sty and all "classic" input encodings such as latin1, do not look at the font encoding, the utf8 one is the only one that does, just to make an initial guess of how much of unicode to cover, – David Carlisle Dec 11 '15 at 18:38

You are saying "it is not forbidden to use fontenc with xelatex." This is true. Actually fontenc is normally used with xelatex as fontspec loads fontenc but not with the T1 option but EU1.

fontenc is a rather special package which can be loaded more than once. In your question you are implicitly assuming that if T1 is loaded it is also the only, the main font encoding of the document. But this here is quite valid too:

\documentclass{article}
\usepackage[T1,LGR,LSF]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{fontspec} % calls \usepackage[EU1]{fontenc}
\begin{document}
abc
\end{document}


What should inputenc do here?

To expand the answer a bit: Documents can load various encodings through fontenc, sometimes even without that the user is aware of it or even wants, e.g. a local class or a (math) package could do it. It is even possible that inputenc is loaded behind the back of the user. It would give quite a mess if inputenc would implement some complicated heuristic to activate a number of chars -- something that normally xelatex users neither need nor want.

• I don't know about LSF. But inputenc could see T1, make the corresponding unicode characters active, and it would see LGR and make the Greek letters Unicode codepoints active and mapped to the corresponding LGR slot. The utf8 letters ασδφγηξ... would use the suitable LGR slot. Is there some major obstacle to that ? – jfbu Dec 11 '15 at 19:15
• Well the main obstacle is that the active encoding at the begin of the document is EU1 so why should the greek be mapped to LGR? Nothing says that LGR will ever be used. – Ulrike Fischer Dec 11 '15 at 19:19
• I see. But it could be a convention to obey the order of the encodings. For example, the latest or earliest encoding wins, if the same utf8 letter can be mapped to various slots in distinct 8bit encoding. EU1 is special here. It seems fontenc has its arms a bit twisted here by fontspec`. – jfbu Dec 11 '15 at 19:26
• inputenc doesn't (and shouldn't) map to "slots". It maps to commands. It is the font encoding files which then map the commands to slots. Beside this: users are quite free to change the encoding in the document. The loading order is a quite weak indiz. If you make chars active you no longer can use them in command names, and it is more difficult to write them to files, so one shouldn't do it without good reasons. – Ulrike Fischer Dec 11 '15 at 19:45
• Well this inputenc certainly can't know. If it should do something in the presence of T1, it would have to activate everything. Every subset must be managed by the user individually anyway. – Ulrike Fischer Dec 11 '15 at 21:22