# What is considered a token for non-standard input encodings?

When I use an input encoding to (La)TeX that allows me to directly input characters such as é and ä (such as by declaring `\usepackage[utf8]{inputenc}`), how is a token defined? Specifically,

• is it possible to write things like `\^ä`; and for this case, would it matter to (La)TeX whether I represented this character
• as U+00E4 or
• with the combining sequence U+0061 U+0308 ,
• or must I use `\^{ä}` ?

(This question was inspired by a comment from user @doncherry to this question on accented characters.)

-
As a further question I'd like to add: What is the definition in Xe/LuaTeX? –  Caramdir Jul 18 '12 at 1:28
@Caramdir It depends from which packages you load. If you don't load `xunicode` (maybe via `fontspec`), there's nothing special in the characters the engines see. And of course, they won't be printed in most cases, as they correspond to inexistent font positions. –  egreg Jul 18 '12 at 9:24

Standard (pdf)LaTeX based on `pdftex` is 8-bit orientated.

If you use an 8-bit encoding like `ansinew`, `ä` is encoded as `11100100`. This means it is 8 bits long and so for `pdftex` it's a single input "entity". In this case `\^ä` can work (if you use `inputenc`, `ä` is a command and what happens depends on the font encoding which sets the actual definition of `ä`).

If you use `utf8`, `ä` is normally encoded as `1100001110100100`. So it is at least 2x8 bits long and `pdftex` sees two input "entities". As `inputenc` makes the first bit active and so `ä` is actually a rather complicated command, a combination like `\^ä` will normally break.

The engines XeTeX and LuaTeX are unicode orientated.

For them a UTF-8 encoded `U+00E4` `ä` is a single entity. With these engines there is no longer a difference between "normal" ASCII and "special" non-ASCII chars: `ä` will be handled like an `a` and `\^ä` will work fine. If the `ä` is encoded as `U+0061 U+0308` then there are also for these engines two input entities. A command like `\^ä` will normally not break in such a case but the output could be different from `\^{ä}`. Again what happens depends on font and on the actual definition of `\^` (normally set by `xunicode` with these engines).

-
I'd add that the `U+0061 U+0308` combinations are not supported by `pdflatex` –  egreg Jul 18 '12 at 13:56
@egreg Where does this restriction derive from? Are all Unicode combining sequences unsupported by `pdflatex`? –  Lover of Structure Jul 19 '12 at 8:56
@user14996: Roughly `inputenc` handles utf8 by making the first octett active (a command) and this commands grabs the following pieces as arguments to build the char. But `inputenc` can't make ASCII chars like `U+0061` active. This would break all command names with an `a` in it. So it would have to define `U+0308` so that it looks back, removes the already printed `a` and replace it by something else: That's not easy. –  Ulrike Fischer Jul 19 '12 at 9:05
@user14996 `pdftex` only works with "one-byte" input. The support for multibyte UTF-8 characters works by looking at the first byte, which bears the information on how many bytes the character is made of, and performing appropriate actions. Supporting also combining characters would require a look ahead that's quite complicated to accomplish in full generality. This is an inherent limitation of eight bit engines, unfortunately. –  egreg Jul 19 '12 at 9:10