Write real UTF32 in Latex?

Question

Is it possible to write a Latex document independent of the input language (Latin, Arabic, Japanese, Hindi)?

I though \usepackage[utf8]{inputenc} was there to do exactly that, but for instance Japanese would not appear when mixed with Latin characters. My current solution is importing the necessary language packages like \usepackage{xeCJK} for Japanese and \usepackage{CJK} for Chinese. Isn't there a way though to define UTF32 and not worry about all the language specific packages like xeCJK and CJK?

Answer 1

XeTeX does have built in support for UTF-32 (and it could be coded in Lua for luatex)

For example

was produced from

\XeTeXinputencoding UTF-32
XXX\XXXdXXXoXXXcXXXuXXXmXXXeXXXnXXXtXXXcXXXlXXXaXXXsXXXsXXX{XXXaXXXrXXXtXXXiXXXcXXXlXXXeXXX}XXX\XXXbXXXeXXXgXXXiXXXnXXX{XXXdXXXoXXXcXXXuXXXmXXXeXXXnXXXtXXX}XXXhXXXeXXXlXXXlXXXoXXX XXXwXXXoXXXrXXXlXXXdXXX\XXXeXXXnXXXdXXX{XXXdXXXoXXXcXXXuXXXmXXXeXXXnXXXtXXX}

in which each X was replaced by a 0 byte, ascii null (control -@) (I had to make that replacement to X just to post to this site, as posting null bytes is tricky)

Very few editors support this format. even emacs only supports utf-8 and utf-16 by default, not utf-32.

In UTF-32 every character takes four bytes and so every ascii character is preceded by three zero bytes, this means that (in contrast to UTF-8) it is tricky to switch between UTF-8 and a legacy encoding as the string \XeTeXinputencoding is represented by a different byte sequence in UTF-32 than it is in UTF-8 or ASCII.

However UTF-8, UTF-16 and UTF-32 are file encodings for the same underlying Unicode character set so whichever of the encodings you use, the same characters can be represented. Whether that character is typeset depends not on the file encoding of the source but rather on the font being used. However you specify the code points for Chinese, if the current font is latin modern, you will get missing glyph warnings as Latin modern does not have those characters.

It is possible using fontspec to specify a different font to be used for different ranges of Unicode, but normally in LaTeX you need more than just a font switch, hyphenation and spacing (and other things) are language specific and best controlled by explicit commands in the document (which can then also select a suitable font) so a mechanism that just switches fonts when it finds a missing glyph would perhaps always produce poor output although at first it might seem an improvement on the current behaviour of no output at all.