My LaTeX text editor is GNU Emacs 25.1.1, which encodes text files in UTF-8. Is there any reason to specify

\usepackage[utf8]{inputenc}

in the preamble? Even if I migrate to a different computer with a different TeX installation, is there any risk that the migrated LaTeX files will be misinterpreted if I leave out this line?

  • 2
    @EvanAad try inserting something like ä or ß... – Skillmon May 17 '17 at 19:59
  • 3
  • 2
    @EvanAad Your conclusion is wrong. – egreg May 17 '17 at 21:50
  • 1
    @EvanAad A UTF-8 encoded file does need inputenc if run with pdflatex; in this case the BOM might give errors. You don't need inputenc for files run with XeLaTeX or LuaLaTeX, provided you load fontspec. – egreg May 17 '17 at 21:53
  • 3
    It isn't strictly true (but not the subject of this question) to say the engine expects ascii it will accept the full byte range the format may or may not be programmed to accept characters at that point. But in any case saying that inputenc is not needed if there is a BOM is entirely misleading for future readers of the site it implies that the BOM is somehow telling latex that the file is utf8 which simply is not the case. if it works with the BOM and no inputenc it would work without BOM and no inputenc and the entire mention of UT-8 is a red herring. You should delete the conclusion. – David Carlisle May 17 '17 at 22:12
up vote 24 down vote accepted

The basic LaTeX/TeX engine expects (or perhaps is meant to process) pure ASCII input. Whenever your file uses any other characters, the inputenc package comes to the rescue, specifying to the engine how to process the symbols you're typing.

So it's quite necessary, whenever you use unicode (non ASCII) characters, to use the inputenc package, in order to have a meaningful output (or sometimes to make a successful run of (La)TeX)

The difference comes with the "naturally UTF8 compliant" engines, such as LuaTeX and XeTeX, which automatically interpret the input files as UTF8 and won't accept different input encodings: in that cases \usepackage[utf8]{inputenc} can be omitted, since it does basically nothing (and is not used internally anyway)

To put it in other terms, the programs do not check whether the file characters comply to the ASCII standards, they simply interpret them to be as such.

  • If I feed a UTF-8-encoded .tex file to a TeX engine that expects pure ASCII, won't it just fail with an error message? – Evan Aad May 17 '17 at 20:06
  • 6
    @EvanAad No, the Byte Order Mark is not mandatory at all (and would actually confuse LaTeX). – egreg May 17 '17 at 21:07
  • 6
    @EvanAad for example wikipedia's utf-8 page says The Unicode Standard neither requires nor recommends the use of the BOM for UTF-8, but does allow the character to be at the start of a file.[34] The presence of the UTF-8 BOM may cause problems with existing software that could otherwise handle UTF-8, for example: – David Carlisle May 17 '17 at 21:11
  • 2
    @EvanAad: No, BOM is strongly discouraged in UTF-8 and is utterly invalid in many contexts (like some programming/script language source files) that might be UTF-8. Basically only some misguided windows-based editors insert BOM in UTF-8 without you manually entering/asking-for it. – R.. May 17 '17 at 21:52
  • 2
    @Davislor: A C source file beginning with a BOM is syntactically invalid. See the language grammar. For better or worse, it's still difficult to use UTF-8 in C sources. As part of (non-wide) string literals it's always worked (practically if not formally, because historically they're just treated as bytes) but otherwise it's not supported by the language without \u/\U or compiler extensions. – R.. May 18 '17 at 14:45

With the 2018 release of LaTeX the test file below produces

enter image description here

as UTF-8 is assumed as the default input encoding unless you specify a different encoding to inputenc and the BOM at the start of the file is handled gracefully (ignored in this case).


Original answer

With inputenc commented out I get

enter image description here

despite typing the input in emacs.

\documentclass{article}

\usepackage[T1]{fontenc}
%\usepackage[utf8]{inputenc}

\begin{document}

© David Carlisle and cost £2000.
\end{document}

Since there seem to be some discussion about the BOM..

If the above file is saved with the byte order mark (or any printable character) before the \documentclass then you get an error

! LaTeX Error: Missing \begin{document}.

but this is not built in to the TeX engine, it is just the default setting of the characters which can be changed depending how you call LaTeX

The commandline

pdflatex '\catcode"EF=9\catcode"BB=9\catcode"BF=9 \input' testfile

would declare the BOM safe and latex would then process the file without error and give the same bad output as shown above. The presence of the BOM in no way implies UTF-8 encoding to the system.

  • Just like you changed the catcode for the UTF-8 BOM, you can change the catcode for the backslash character, and then the file won't compile even if it contains the \inputenc line. – Evan Aad May 17 '17 at 22:54
  • @EvanAad the interpretation of every byte is setable within tex or even as I show here in the call to a particular file, your assertion that the engine is restricted to ascii is false even by default and the assertion that a leading byte order mark necessarily gives an error due to the engine's ascii nature is also false. It is just a feature of the tex assignments (which could be in the format or on the commandline) many editors do assignments like this on the commandline routinely when they run tex in the background. – David Carlisle May 17 '17 at 22:57
  • @EvanAad making \ not be the escape character would just be peverse but a commandline such as I show above would be a perfectly reasonable way to set up emacs tex command if you had to share files with someone using an editor that insisted on adding a BOM. You can make it safe, but it does nothing. – David Carlisle May 17 '17 at 23:03
  • FWIW, the combination of BOM / UTF-8 is discouraged anyway. A BOM only makes sense with UTF-16 / UTF-32 encodings, as UTF-8 does not suffer from byte ordering issues in the first place. If your editor allows it, make it omit the BOM for UTF-8 output. – DevSolar May 18 '17 at 8:01
  • @DevSolar yes as was explained at length in comments under the original question although I have since deleted most of that comment thread. – David Carlisle May 18 '17 at 8:03

Here are some examples to make explicit some detail (implicit in the other answers), which may help clear up any remaining confusion.

Consider the following Unicode text añ©ⱥ which consists of:

A byte is a number from 0 to 255 in decimal, or 00 to FF in hexadecimal. So when encoded with UTF-8, the above "four-character" string corresponds to, in the file, the 8 bytes 61 C3 B1 C2 A9 E2 B1 A5.

tex/pdftex WITHOUT inputenc

The engine sees the input as a stream of bytes (8 bytes in the above example). It considers each of them as as a character, and decides to typeset the corresponding character from either the T1 (Cork) encoding or OT1 encoding (the default) or whatever is set up. Examples:

OT1 Above, OT1 has no characters for those bytes so nothing gets typeset. T1 I hope you can see what's happening: each of the 8 bytes is treated as a character and output: e.g. the byte C3 is “Ô in T1.

tex/pdftex WITH inputenc

With \usepackage[utf8]{inputenc}, TeX correctly sees every sequence of UTF-8 bytes as a Unicode character. For example, when TeX sees the byte sequence C3 B1, it understands that you mean the Unicode character U+00F1. (The way this is done is that bytes larger than 127 (80 to FF in hexadecimal) are set up to be active characters that expect further input — this is possible because of a useful design of UTF-8. See texdoc utf8ienc for details.)

TeX still needs to know what to do with that Unicode character. A big bunch of definitions (such as \DeclareUnicodeCharacter{00F1}{\~n} saying what to do with the character U+00F1) are included in the TeX distribution (file texmf-dist/tex/latex/base/utf8.def on TeX Live). So using \usepackage[utf8]{inputenc} will help if your characters have such definitions (again, see texdoc utf8ienc for the full list), or if you're willing to define them yourself.

inputenc

With a Unicode-aware engine (XeTeX or LuaTeX)

You don't need inputenc. The engine will expect UTF-8 (by default), and understand the input simply as Unicode characters, and for each of them it simply typesets that character from the currently selected font.

xetex

What about BOM?

With UTF-8 the BOM (byte order mark) isn't needed (it was meant for non-byte-oriented encodings, like UTF-16 and UTF-32), and is strongly discouraged. Typical “good” editors won't include it. Just forget about it; you aren't likely to encounter it in practice.

But if somehow your file does end up including it, then it's just a sequence of bytes EF BB BF (the UTF-8 encoding of U+FEFF), and I think you have enough information above to work out what would happen if those bytes were present in the file at what place.

What if my file contains only "normal" characters?

If you mean Latin-script characters without accents, then UTF-8 has the property that it coincides with ASCII on the range 0 to 127 (00 to 7F). So a file containing only those characters, encoded in UTF-8, is indistinguishable from one encoded in ASCII. Naturally, the output is identical too.

There are two types of TeX engines: ones that expect UTF-8 (e.g. LuaTeX and XeTeX), and ones that don't (e.g. pdftex).

If a TeX engine that expects UTF-8 is fed a UTF-8-encoded .tex file, the \usepackage[utf8]{inputenc} command can be omitted. In fact, a warning will be issued if it's not.

If a TeX engine doesn't expect UTF-8, and the TeX file contains non-ASCII characters, and the file doesn't contain a suitable inputenc command, strange output may result, as demonstrated by David's answer.

  • 1
    A UTF-8 file starting with a BOM and processed with pdflatex will raise errors and compile wrongly. You need nothing else than the BOM to make it fail, so whether you load inputenc or not is actually irrelevant: this is true, but completely useless information. – egreg May 17 '17 at 22:23
  • 1
    This answer is entirely misleading. If the file is ascii then inputenc is not needed. If you are using luatex or xelatex then inputenc will generate a warning but otherwise do nothing. If you are using pdflatex or latex and the file uses any utf8 encoding other than the trivial ascii subset then inputenc is needed. the BOM is unrelated to inputenc it may (like any character before \documentclass give errors) – David Carlisle May 17 '17 at 22:24
  • 2
    No. Any normal reader who reads A UTF-8 encoded .tex file does not require the \usepackage[utf8]{inputenc} if it starts with a BOM or if only ASCII characters were encoded in it. would read that as saying that you can use a BOM as an alternative to inputenc. You may as well say that you do not need inputenc if there is a X before \documentclass, it is exactly the same. the fact that it is a BOM is completely irrelevant – David Carlisle May 17 '17 at 22:29
  • 1
    if pdftex only expected "pure ascii" then the output from my answer would not include non-ascii characters such as the accented A. – David Carlisle May 17 '17 at 22:32
  • 1
    it's better after the edit but still gives bad advice. In the second paragraph can be omitted, as it has no effect. should say should be omitted as otherwise it gives a warning message in the third paragraph its behavior is undefined is wrong, there is no behaviour that is undefined or system dependent here, some behaviours may give errors or be otherwise not useful but the behaviour is defined in all cases. It's not true that the file has to be ascii up to that point for example \usepackage{ÉÉÉ} should work, or you could have comments that were not ascii. – David Carlisle May 17 '17 at 23:58

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.