5

I have a LaTeX document in UTF-8 encoding:

\documentclass{article}
\usepackage[utf8]{inputenc}         
\usepackage[T1]{fontenc}           

\begin{document}
  äääääääääääääääööööööööööööööööööööööüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüääääääääääääääääääääääääääääääääääüüüüüüüüüüüüüü
\end{document}

When compiling this with pdflatex on MacOS X in a terminal using an UTF-8 locale, the (expected) overfull \hbox warning comes out in Latin1 encoding – in the terminal as well as in the log file:

This is pdfTeX, Version 3.1415926-2.5-1.40.14 (TeX Live 2013/MacPorts 2013_1) (format=pdflatex 2013.7.29)  2 SEP 2013 17:28
entering extended mode
 restricted \write18 enabled.
 %&-line parsing enabled.
**test

<SNIP>

Overfull \hbox (419.53918pt too wide) in paragraph at lines 7--8
[]\T1/cmr/m/n/10 äääääääääääääääööööööööööööööööööööööüüüüüüüüüüüüüüüüüüüüüüüüü
üüüüüüüüüüüüüüüüüüüüüüüüüüüüüüüääääääääääääääääääääääääääääääääääüüüüüüüüüüüüüü

 []

[1

{/opt/local/var/db/texmf/fonts/map/pdftex/updmap/pdftex.map}] (./test.aux) ) 
Here is how much of TeX's memory you used:
 617 strings out of 494181
 5593 string characters out of 6156676
 52409 words of memory out of 5000000
 3964 multiletter control sequences out of 15000+600000
 4403 words of font info for 15 fonts, out of 8000000 for 9000
 470 hyphenation exceptions out of 8191
 23i,4n,17p,299b,113s stack positions out of 5000i,500n,10000p,200000b,80000s
{/o
pt/local/share/texmf-texlive/fonts/enc/dvips/cm-super/cm-super-t1.enc}</opt/loc
al/share/texmf-texlive/fonts/type1/public/cm-super/sfrm1000.pfb>
Output written on test.pdf (1 page, 12195 bytes).
PDF statistics:
 13 PDF objects out of 1000 (max. 8388607)
 8 compressed objects within 1 object stream
 0 named destinations out of 1000 (max. 500000)
 1 words of extra memory for PDF output out of 10000 (max. 10000000)

I am using a post-filter program for the log file (namely pydflatex by Olivier Verdier) that expects UTF-8 encoded input and, thus, breaks at this line.

So what determines the encoding of the LaTeX log file – and how can I change it?

  • It's not Latin-1 encoded, actually: the log shows the character slots after the translation made by fontenc, which has many points in common with Latin-1, but it's T1 in your case. – egreg Sep 2 '13 at 15:45
  • I don't think you can easily with pdftex (ie without changing the source and recompiling from scratch) the output there isn't really any encoding at all it is just the bytes used in the font encoding which happen to be T1 in that case which happens to be similar to latin1 in that range. You might prefer to use xetex or luatex rather than pdftex – David Carlisle Sep 2 '13 at 15:49
  • oh @egreg got there first:-) – David Carlisle Sep 2 '13 at 15:49
  • @DavidCarlisle: Ah, my VIM editor told me it to be Latin-1, it never occurred to me that it could be the font encoding. So a log-file parser for LaTeX has to be able to deal with basically arbitrary sequences of bytes. – Daniel Sep 2 '13 at 15:51
  • yes, try some math in your overfull box for an extreme example – David Carlisle Sep 2 '13 at 15:57
11

Unicode input with LaTeX (as opposed to LuaLaTeX and XeLaTeX) doesn't survive the internal processing, in particular it's lost when messages regarding overfull boxes are shown.

What those messages show is a representation of the characters actually used for typesetting, which are no more Unicode, but correspond to the slots in the current output encoding.

So, for example, ä becomes first \"a because of

\DeclareUnicodeCharacter{00E4}{\"a}

in utf8enc.dfu and then the macros related to fontenc with the T1 encoding translate it into “print character in slot 0xE4”, because of

\DeclareTextComposite{\"}{T1}{a}{228}

in t1enc.def (and decimal 228 is hexadecimal 0xE4). This is actually the same slot for ä in Latin-1, but the T1 encoding differs from Latin-1; for instance ß sits at position 255 = 0xFF in T1, while it's in slot 223 = 0xDF in Latin-1.

What you need is a filter translating back from T1 (or another output encoding) to Unicode. The filter should also take into account the font used, which has the output encoding in its name: not a trivial task, in particular for math symbols.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.