After switching to OS X one of the first things I had to learn the hard way is that many non-ASCII characters, such as the German ü can be encoded in (at least) two different forms in UTF8:


(The glory details are all described here)

Basically, all operating systems and applications today use NFC only, with the exception of Mac OS X, in which some applications (e.g., OpenOffice or the HFS+ file system) use NFD. The result is that if you copy & paste some text from such an application (e.g., the output of the ls command) into your LaTeX document, everything looks fine.

\usepackage[utf8]{inputenc} % comment out for lualatex/xelatex
\usepackage[T1]{fontenc}    % comment out for lualatex/xelatex


However, when compiling with pdflatex:

! Package inputenc Error: Unicode char \u8:̈ not set up for use with LaTeX.

A often given answer with respect to unicode problems is "use lualatex/xelatex". However, that does not seem to help here either. If compiling with lualatex/xelatex, the output does not contain the umlauts:

enter image description here

Question: The inputenc package with [utf8] is apparently not able not handle NFD. Is it possible to extend it so that the above does compile?


Note that the MWE, if copied & pasted from here into a new document, actually does compile. Apparently either my browser or the SE site transparently transforms NFD to NFC. (For Safari and Crome that seems to be the case indeed; I have also tried Firefox without success). I have yet to figure out how to provide some piece of text in NFD here.

Excursus: A Bit of Extra Background on HFS+

I first stumbled over this issue when trying to put the output of a ls command into my LaTeX document: The source of many, many problems in OS X is that the HFS+ file system uses (for some totally weird reasons) NFD. Even worse: HFS+ transparently transforms all NFC characters it gets as input into NFD internally. Practically, this means that the filenames you get out are different than those you have put in: If you create a file ü (the keyboard delivers NFC) and then list the directory (the file system delivers NFD) , the name looks same, but in fact is different. A short illustration test (executed in an empty dir):

$ echo ü; echo ü | xxd; touch ü; ls; ls | xxd
0000000: c3bc 0a                                  ...
0000000: 75cc 880a                                u...

This is the reason so many tools (unison, svn, git, ...) or bash's tab completion choke on OS X on filenames containing umlauts – and that you cannot use the output of ls directly in your LaTeX document.

  • 1
    This NFC/NFD HFS+ thing keeps biting me every time I use ls. It's really annoying.
    – You
    Commented Jan 18, 2013 at 23:18
  • @You: Apply actually does call it a "feature", but everybody else considers it as a serious bug.
    – Daniel
    Commented Jan 19, 2013 at 8:45
  • Of course the HFS+ filesystem features have nothing to do with the problem at hand, do they?
    – egreg
    Commented Jan 19, 2013 at 9:44
  • @egreg: Well, they originally were the source of my problems: I tried to embed the output of ls into my LaTeX document. They are probably also the instance on which OS X users sooner or later will hit the root of the problem. However, rephrased the excursus to make this more clear.
    – Daniel
    Commented Jan 19, 2013 at 11:58
  • 2
    Or you can use \XeTeXinputnormalization=1 to normalize input into NFC. Commented Jan 19, 2013 at 14:38

5 Answers 5


(see possible solutions at the end.)

A survey of NFC and NFD UTF-8 forms in XeLaTeX input

xelatex almost handles NFD form almost out-of-the-box. You will need to load the xltxtra package, which you probably always want to load when using XeLaTeX, anyway.

Here's an example bash-script to create a test document (mkutest.sh):

#! /bin/bash
  cat <<'EOF'
  uconv -f utf-8 -t utf-8 -x nfc <<<"UTF-8-NFC: $TEXT"
  uconv -f utf-8 -t utf-8 -x nfd <<<"UTF-8-NFD: $TEXT"
  cat <<'EOF'
) > utest.tex

This script uses uconv (from ICU, See note 1 below) to create the two representations (NFC and NFD) of the same text and adds the XeLaTeX pre-/post-amble. This script should be "safe" to copy from the web page, since it uses the converter and the text input to it can be in any UTF-8 form. (See note 2 below for a version that does not depend on uconv.)

The created file looks like this (utest.tex):


UTF-8-NFC: åäöüÅÄÖÜß

UTF-8-NFD: åäöüÅÄÖÜß


(Note: This may not yield the desired file if just copied from the web. See the warning in the question.)

The result of running this through XeLaTeX is a PDF with the text:

enter image description here

where the two lines does not look exactly the same (even apart from the label). The accents in the first line look OK, but the accents of the capital letters in the second line are vastly misaligned.

So, although XeLaTeX can handle NFD form, it may not do it properly...

If \usepackage{xltxtra} is omitted the PDF looks like:

example without the xltxtra package

which corroborates the example use of XeLaTeX in the question. Furthermore: Note that nothing at all shows up in the first row and the ß is missing on the second row. This is because the loaded fonts don't have the glyphs to render this. The xltxtra loads the package fontspec, which by default loads the font "Latin Modern". Without this only legacy fonts are loaded, which does not at all play nice with unicode text.

I have tested with different fonts (system fonts loaded with the fontspec command \setmainfont{<name of font>}). The result have been somewhat diverse. For all fonts that have the needed glyphs the first line looks correct. The second line, however, can come out in some different forms. For example with the accents after the base letters, as if they were non-combining; or with missing-glyph-boxes after the base letters...

As Khaled noted, XeTeX can normalize its input to NFC. Adding \XeTeXinputnormalization=1 to the preamble, before any non NFC-text is read, and still using \usepackage{xltxtra} and/or other means to set up proper fonts, the output is:

example with automatic NFC-normalization

This time the two lines does look exactly the same (apart from the label).

What to do?

If using XeTeX, \XeTeXinputnormalization=1 is definitely a solution. Just remember that you have to properly set up the fonts.

The other way to go, which works with all(?) programs that support UTF-8 NFC text input, is to convert the input files beforehand.

To massage the files into NFC form one can, for example, use uconv (from ICUSee note 1 below) as I did in the MWE-generator above.

$ uconv -o outfile.tex -f utf-8 -t utf-8 -x nfc infile.tex

(This works with UTF-16 encoding -- and others -- too. Just change the from (-f) and to (-t) options appropriately.)

Disclamer: Use this command at your own risk. Be sure to keep the original file until you can verify the result.

This should probably be safe to run on any (7-bit) ASCII or UTF-8 encoded tex file. If the file is already in NFC the conversion should not change anything, since it is idempotent. Files containing only 7-bit ascii are already in NFC, since 7-bit ASCII is a subset of UTF-8 and contains no combining characters that could make the text non-NFC.


  1. The uconv utility from ICU is in the package libiuc-dev on my Ubuntu 12.04 64-bit.
    (I think it is among the examples for the ICU4C library, but I could not find any info about the it from a quick search on the homepage. I'm a bit confused...)

  2. As requested by David in his comment I have made a version of the MWE-generator that does not depend on uconv.

      echo '\documentclass{article}'
      echo '\usepackage{xltxtra}'
      echo '\begin{document}'
      echo -e 'UTF-8-NFC: \xc3\xa5\xc3\xa4\xc3\xb6\xc3\xbc\xc3\x85\xc3\x84\xc3\x96\xc3\x9c\xc3\x9f'
      echo -e 'UTF-8-NFD: \x61\xcc\x8a\x61\xcc\x88\x6f\xcc\x88\x75\xcc\x88\x41\xcc\x8a\x41\xcc\x88\x4f\xcc\x88\x55\xcc\x88\xc3\x9f'
      echo '\end{document}'
    ) > utest.tex

    This version only depends on that echo -e interprets \xHH (and that echo without -e does not).

    I kept the other version (above, in the main text) since it allows for easy changes in the sample text.

    For the interested, the hex escapes are generated by uconv -x '[:Cc:]>; ::nfc;' <<<"$TEXT" | hexdump -v -e '/1 "%02x "' | sed -e 's/[[:xdigit:]][[:xdigit:]]/\\x\0/g; s/ //g' for NFC, &sim. for NFD.

  • I wanted to put images of the output in my post, but I'm a new user, so I don't have enough rep.
    – Johan_E
    Commented Jan 19, 2013 at 15:58
  • You can insert images with the provided interface; just remove the ! in front of the obtained internal link and a user with enough privileges will add it back.
    – egreg
    Commented Jan 19, 2013 at 16:08
  • @JohanE: +1 and many thanks for the "MWE generator" :-) Today, I am also using uconv for the transformation (see my answer), which, however, is not out of the box available on OSX.
    – Daniel
    Commented Jan 19, 2013 at 16:34
  • Great edit, the answer is getting better and better! One suggestion for the "MWE generation": As uconv is not trivially available for OSX you might instead use xxd to produce a hex-dump for NFC and NFD; include this in your bash script and use xxd -r there to generate the NFC/NFD strings. xxd is a pretty old (common) UNIX tool and also part of a vanilla OSX installation.
    – Daniel
    Commented Jan 29, 2013 at 15:36
  • OK, I added a version of the script that does not use uconv. Instead it just use echo -e with hex escapes.
    – Johan_E
    Commented Jan 29, 2013 at 18:41

It's really difficult to support combining characters in inputenc. The package works by making all non-ascii characters "active" with a macro definition. For T1 and similar 8 bit encoding this definition just directly typesets the character in that slot, but for UTF-8, it triggers a little parser looking ahead for the next few bytes of the UTF8 multi-byte encoding to work out which character is needed. Classic TeX typesets a character token as soon as it finds one and places it on the horizontal list being constructed. The important thing to note though is that horizontal lists can not be de-constructed there is no \lastchar primitive analogue of \lastskip or \unskip. This means that by the time you find the combining accent character the base has been typeset into the list and become inaccessible.

There are some TeX tricks that you can play in some cases to get TeX to scan ahead and find these things but they are inherently fragile and tend to break any other loaded package. It is a lot more robust to pass the file through a Unicode normalisation tool and get it back into NFC form.

Or of course use a unicode aware TeX such as luatex or xetex.

  • Hm... lualatex or xelatex do not seem be able to deal with NFD either, at least not out of the box. See edited question.
    – Daniel
    Commented Jan 19, 2013 at 8:46
  • @Daniel For xe/luatex do what Khaled says, he knows about that stuff:-) Commented Jan 19, 2013 at 14:50

Apparently, there is no easy solution to this problem within LaTeX itself. So if one does not want to use XeLaTeX with \XeTeXinputnormalization=1, as suggested by Khaled, the best approach is probably to normalize the input (source) to NFC, as suggested by David.

This is what I have been doing in the past. It is, however, less easy as it sounds: There are surprisingly few tools available that actually can do this kind of normalization. In particular iconv and the system library behind it, which is used by most editors (including vim) for encoding transformations, is not NFD capable. So in the following I would like to present tools and approaches to do the conversion. I make this answer community wiki, so please contribute to it if you know more tools.

The Poor Man's Approach (OS X): TextEdit

  • Open the document (or just paste the NFD text) in TextEdit
  • Set TextEdit to plain text format (Format->Make Plain Text)
  • Save the document in some other encoding (not UTF-8) that contains the characters represented in NFD. (For instance for German umlauts you choose Western (ISO Latin 9)).
  • Save the document again in UTF-8 encoding (or just copy the text from TextEdit into your LaTeX document). The resulting encoding is now NFC.

The advantage of this approach is that it works without further third-party tools. It requires, however, some manual steps and the transition over some intermediate encoding implies the danger of loosing uncommon characters.

Real Conversion Tools


The uconv utility is the most complete converter I have found so far. It actually is an example application of the ICU project, an extensive set of open-source libraries and APIs to deal with unicode issues that is targeted at software developers. It seems to be available in some Linux distros, but unfortunately is not yet part of OS X nor MacPorts. So you have to download and compile it from the sources, which, however, works flawlessly.

Update March 2015: Meanwhile, uconv can be obtained easily for most platforms: On MacPorts by installing the package icu, Homebrew users install icu4c, and on Ubuntu the package is named icu-devtools.

Once, uconf is available, you can use it like iconv, with the exception that you have an extra -x command line option to specify the normalization form. The following, executed in a bash shell, converts paper.tex to paper-nfc.tex:

uconv -f utf-8 -t utf-8 -x NFC paper.tex >paper-nfc.tex

A useful one-liner for OS X is the following. It recodes the content of the clipboard to NFC:

pbpaste | uconv -f utf-8 -t utf-8 -x NFC | pbcopy

Today I use this when copying content, such as the output of ls or text blocks from an OpenOffice document into my LaTeX document. Just execute the above line after copying and before pasting and everything is fine. (For some extra comfort, one might create and Automator workflow that runs this line and assign it a keyboard shortcut to the workflow.)

Online Normalization Tools

Yet another possibility would be to use some online web service for transformations. However, so far I have found only this one:

  • Thanks for this solution. uconv on OS X is available via homebrew: brew install icu4c
    – tom
    Commented Apr 3, 2014 at 7:55

I want to supplement the David's answer. He wrote that there is no possibility to deal with NFD utf8 codes in 8bit TeX (i.e. in pdfTeX) because the base character is set to horizontal mode and it is not accessible when we read the next code.

This is not exactly true because pdfTeX includes encTeX extension. This extension is able to declare utf8 codes as one byte or one control sequence. It reads byte sequences and converts them to control sequences at input processor level. CSplain activates encTeX during format generation and reads the table with the basic set of NFC utf8 codes. It pre-loads the 8bit font with all characters mentioned in the example here and the NFC utf8 codes are mapped to the character slots in the font. So, we can simply write åäöüÅÄÖÜß and we get åäöüÅÄÖÜß. The DFC codes are not prepared by csplain but they can be simply added by the following code.

% use csplain format
% syntax: \mubyte control-sequence byte-sequence\endmubyte
\mubyte\nfdaring     a^^cc^^8a\endmubyte \let\nfdaring=å
\mubyte\nfdadieresis a^^cc^^88\endmubyte \let\nfdadieresis=ä
\mubyte\nfdodieresis o^^cc^^88\endmubyte \let\nfdodieresis=ö
\mubyte\nfdudieresis u^^cc^^88\endmubyte \let\nfdudieresis=ü
\mubyte\nfdAring     A^^cc^^8a\endmubyte \let\nfdAring=Å
\mubyte\nfdAdieresis A^^cc^^88\endmubyte \let\nfdAdieresis=Ä
\mubyte\nfdOdieresis O^^cc^^88\endmubyte \let\nfdOdieresis=Ö
\mubyte\nfdUdieresis U^^cc^^88\endmubyte \let\nfdUdieresis=Ü

UTF-8-NFC: åäöüÅÄÖÜß

UTF-8-NFD: åäöüÅÄÖÜß % NFD codes generated by the uconv, see above the accepted answer


Note, that the utf8 codes used in \let command arguments are normal NFC codes. They are re-interpreded by encTeX as one byte (one slot in the preloaded 8-bit font).

Note2: the byte sequence for DFC codes begins by the base character. If this base character follows the next declared bytes then this byte sequence is transformed by encTeX to the declared control sequence. If the base character isn't followed by the declared code then it is treated as normal single character (inside control sequences or as a simple character to be typeset).

  • That looks promising. However, could you also explain how to apply this the the MWE from the question? (I just got a undefined control sequence \mubyte).
    – Daniel
    Commented Apr 1, 2015 at 13:31
  • @Daniel The first comment in my MWE says: use csplain format. It means, you cannot use LaTeX, for example, because LaTeX doesn't activate encTeX extension during format generation. My answer only adds the information that this is possible in 8bit TeX engines, but (of course) this is impossible in normal LaTeX, unless LaTeX is re-generated with encTeX extension activated and inputenc/fontenc processing is totally changed.
    – wipet
    Commented Apr 1, 2015 at 13:47
  • Ah, thanks for the clarification! As a "typical (even though experienced) LaTeX user" I have to shamefully admit that I have never heard of csplain format before and don't have a clue what format generation is. I kindly suggest to edit the answer so that the applicability becomes clear also for the average LaTeX user.
    – Daniel
    Commented Apr 1, 2015 at 15:36

In case anyone writes the documents with python. Make sure you save it as UTF-8:

with open(f'{filename}.tex', 'w', encoding='utf-8') as tex_file:
  • Welcome to tex.sx. Commented Feb 8 at 15:05
  • thanks, great to be here :)
    – Marco
    Commented Feb 9 at 9:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .