2

I usually use newtxtext for most of my documents, but I'm trying to switch to XeTeX from pdfLaTeX. The Unicode characters ¿ and ¡ show up as £ and ą when using newtxtext on LuaTex and XeTeX, so I just use fontspec now. Note that this doesn't happen if I enter the characters with the standard commands like \textquestiondown. I assume this is just some sort of incompatibility with the new engines, although I haven't read anywhere that they shouldn't be used in these.

[1]

Likewise, I'm also following the practice of using polyglossia instead of babel because I've read polyglossia is designed as the successor for babel in XeTeX, but is there any real reason not to keep using babel on the new engines? Why is there two big packages for the same task?

Thankfully I doubt I would have any big problems or incompatibilities caused by Spanish localization, since it's just Latin script with diacritics already supported in TeX, but I can imagine CJK-languages would be more prone to problems like this.

Small example of newtxtext not working right on XeTeX or LuaTeX:

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{newtxtext}
\begin{document}
«¡¿Por—qué?!»

\guillemotleft\textexclamdown\textquestiondown{}Por---qu\'e?!\guillemotright
\end{document}
  • 1
    Have a look at the documentation for newtx on CTAN. It includes examples of how to set the package up for XeLaTeX and LuaLaTeX in Section 5 on page 10. – John May 31 '18 at 1:39
  • 1
    Could you post a small document that gives us something to start from? – Teepeemm May 31 '18 at 1:47
  • 2
    You can use Babel. Polyglossia doesn't support all languages supported by Babel (yet?). Polyglossia requires Xe/LuaTeX, so isn't usable with older engines. You seem to know all this already, so surely it is obvious why there are two packages: they don't do the same thing, even though they may serve the same purpose in some cases. Using standard text font packages is not recommended. (If you have to ask if it is OK, you shouldn't be doing it.) But standard maths packages are fine. For text/maths packages, care is needed as the linked manual shows. – cfr May 31 '18 at 2:23
  • 2
    Not all font packages are a bad idea. Some, like libertine, detect if running under XeLaTeX or LuaLaTeX and use fontspec and system fonts in that case. But you can't always know if a package does this. AFAICS newtxtext does not do special things for Xe/LuaLaTeX and should probably not be used, newtxmath on the other hand can do stuff for these two engines and can be used with the appropriate settings. You also don't load fontenc and inputenc with Xe/LuaLaTeX, but that doesn't change anything here. BTW: netxtext is based on the open type font TeXGyre Termes so you might try that – moewe May 31 '18 at 5:52
  • 2
    Re babel v polyglossia. There are some things polyglossia does better than babel, especially for languages with non-Latin scripts (I'm thinking of RTL languages like Arabic, Hebrew, ...). And there are some things that babel still does better than polyglossia. Amongst those is the interface for other packages. At the moment polyglossia is only occasionally maintained and has many open bug reports and feature requests, while babel is actively being developed (with progress on the RTL front). If babel worked for you before there is no real reason to switch to polyglossia. – moewe May 31 '18 at 5:58
5
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{newtxtext}

never use [T1]{fontenc} with xetex or luatex this specifies that you want to use a legacy tex-specific 8-bit font encoding, so undoes almost all of xetex's Unicode support. You will get incorrect characters (as you saw) and incorrect hyphenation for any non-ascii text even if the characters appear correct.

[utf8]{inputenc} does nothing in luatex or xetex other than issue a warning that it shouldn't be used.

Font packages such as newtxtext mostly shouldn't be used as mostly font packages for pdflatex are setting up the fonts that have been re-encoded for TeX in encodings such as T1, but some are OK to use as they detect the TU (unicode0 encoding and load Unicode (typically OpenType) versions of the fonts. As I just noticed moewe said in comments using TeXGyre Termes via fontspec effectively gives you the OpenType base font on which the textext fonts were based.

babel is fine, polyglossia is/was developed as an alternative to babel but currently babel is more actively maintained, so try both and see which you prefer.

  • Thanks. This sums up my concerns. I've been switching to polyglossia, but since people here seem to mention it's perfectly fine to use babel, I may have to switch back, after all, the Spanish support IS better there. It's an infortunate situation that something like UTF-8 (or Unicode) support is so hard to get working on older TeX engines, but I assume it's something way too intrinsic with the inners-workings of the engines. – ficion May 31 '18 at 20:40
  • @ficion the restriction to a maximum of at most 256 characters per font is built into the core of tex and built in to its syntax for almost all core commands and character handling such as \char and \mathchardef so supporting a Unicode font is necessarily a breaking change. There is a reason luatex and xetex are new systems not just an updated (pdf)tex. – David Carlisle May 31 '18 at 21:18
2

To explain somewhat why the output differs, consider this example, using only the ¿.

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\begin{document}

\tracingmacros1
¿
\tracingmacros0

\showoutput
\end{document}

If you compile with XeLaTeX, the log contains no trace of macro expansion. And the \showoutput contains a line

....\T1/lmr/m/n/10 ¿

If you compile with PDFLaTeX, the log contains this (as explained below the log file is considered iso-latin-1 encoded in that case by Emacs and decoded as such, but what is pasted here is the utf-8 encoded conversion)

Â->\UTFviii@two@octets Â

\UTFviii@two@octets #1#2->\expandafter \UTFviii@defined \csname u8:#1\string #2
\endcsname 
#1<-Â
#2<-¿

\UTFviii@defined #1->\ifx #1\relax \if \relax \expandafter \UTFviii@checkseq \s
tring #1\relax \relax \UTFviii@undefined@err {#1}\else \PackageError {inputenc}
{Invalid UTF-8 byte sequence}\UTFviii@invalid@help \fi \else \expandafter #1\fi

#1<-\u8:¿ 

\u8:¿ ->\IeC {\textquestiondown }

\IeC ->\ifx \protect \@typeset@protect \expandafter \@firstofone \else \noexpan
d \IeC \fi 

\@firstofone #1->#1
#1<-\textquestiondown 

\textquestiondown ->\T1-cmd \textquestiondown \T1\textquestiondown 

\T1-cmd #1->\ifx \protect \@typeset@protect \@inmathwarn #1\else \noexpand #1\e
xpandafter \@gobble \fi 
#1<-\textquestiondown 

\@inmathwarn #1->\ifmmode \@latex@warning {Command \protect #1 invalid in math 
mode}\fi 
#1<-\textquestiondown 

and one finds also

....\T1/cmr/m/n/10 ¾

in the log.

About the weird ¾ it is because my Emacs/AUCTeX considers the log file to be iso-latin-1 encoded and this is character at slot 190. The buffer from which I copied pasted actually uses UTF-8: buffer code: #xC2 #xBE but file code: #xBE (encoded by coding system iso-latin-1-unix).

On the other hand ¿ has UTF-8 representation file code: #xC2 #xBF (encoded by coding system utf-8-unix), and we see above how this creates a lot of agitation at (PDF)LaTeX+inputenc[utf8] side to convert the 0xBF (191) to 0xBE(190) which matches the actual location in T1 font encoding.

On the other hand in the XeLaTeX case, inputenc does nothing in case of utf8 and as a result the 191 remains 191 and we get to see in PDF the glyph which happens to be at slot location 191 in T1-font encoding.

Notice also that if we query the meaning of \T1\textquestiondown in the PDFLaTeX case via

\expandafter\show\csname T1\string\textquestiondown\endcsname

we get this in log:

> \T1\textquestiondown=\char"BE.
<recently read> \T1\textquestiondown 

l.16 ...sname T1\string\textquestiondown\endcsname

where we find indeed the 0xBE=190 which is T1 slot for ¿.

Related: Why does inputenc abandon so quickly under "utf8 based engines"?

  • Thanks a lot for this insightful answer, this was what I was actually wondering about fontenc on XeTeX, as I thought it'd try to do the same work as in pdfTeX. I'm currently trying to learn more about how macros work inside, so this is very helpful; thanks again. – ficion May 31 '18 at 20:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.