13

When I use an input encoding to (La)TeX that allows me to directly input characters such as é and ä (such as by declaring \usepackage[utf8]{inputenc}), how is a token defined? Specifically,

  • is it possible to write things like \^ä; and for this case, would it matter to (La)TeX whether I represented this character
    • as U+00E4 or
    • with the combining sequence U+0061 U+0308 ,
  • or must I use \^{ä} ?

(This question was inspired by a comment from user @doncherry to this question on accented characters.)

2
  • As a further question I'd like to add: What is the definition in Xe/LuaTeX?
    – Caramdir
    Jul 18, 2012 at 1:28
  • @Caramdir It depends from which packages you load. If you don't load xunicode (maybe via fontspec), there's nothing special in the characters the engines see. And of course, they won't be printed in most cases, as they correspond to inexistent font positions.
    – egreg
    Jul 18, 2012 at 9:24

1 Answer 1

10

Standard (pdf)LaTeX based on pdftex is 8-bit orientated.

If you use an 8-bit encoding like ansinew, ä is encoded as 11100100. This means it is 8 bits long and so for pdftex it's a single input "entity". In this case \^ä can work (if you use inputenc, ä is a command and what happens depends on the font encoding which sets the actual definition of ä).

If you use utf8, ä is normally encoded as 1100001110100100. So it is at least 2x8 bits long and pdftex sees two input "entities". As inputenc makes the first bit active and so ä is actually a rather complicated command, a combination like \^ä will normally break.

The engines XeTeX and LuaTeX are unicode orientated.

For them a UTF-8 encoded U+00E4 ä is a single entity. With these engines there is no longer a difference between "normal" ASCII and "special" non-ASCII chars: ä will be handled like an a and \^ä will work fine. If the ä is encoded as U+0061 U+0308 then there are also for these engines two input entities. A command like \^ä will normally not break in such a case but the output could be different from \^{ä}. Again what happens depends on font and on the actual definition of \^ (normally set by xunicode with these engines).

4
  • 1
    I'd add that the U+0061 U+0308 combinations are not supported by pdflatex
    – egreg
    Jul 18, 2012 at 13:56
  • @egreg Where does this restriction derive from? Are all Unicode combining sequences unsupported by pdflatex? Jul 19, 2012 at 8:56
  • 1
    @user14996: Roughly inputenc handles utf8 by making the first octett active (a command) and this commands grabs the following pieces as arguments to build the char. But inputenc can't make ASCII chars like U+0061 active. This would break all command names with an a in it. So it would have to define U+0308 so that it looks back, removes the already printed a and replace it by something else: That's not easy. Jul 19, 2012 at 9:05
  • 2
    @user14996 pdftex only works with "one-byte" input. The support for multibyte UTF-8 characters works by looking at the first byte, which bears the information on how many bytes the character is made of, and performing appropriate actions. Supporting also combining characters would require a look ahead that's quite complicated to accomplish in full generality. This is an inherent limitation of eight bit engines, unfortunately.
    – egreg
    Jul 19, 2012 at 9:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .