# Problem with XeTeX input and output encoding

As we all known, the TeX engine XeTeX takes UTF-8 as its default input and output encoding format. However, it provides two new primitive control sequences that could define input encoding:

• \XeTeXinputencoding defines the input encoding of the following text.
• \XeTeXdefaultencoding defines the input encoding of subsequent files to be read.

Thus, one could put one of these two commands to specify the encoding that will be read by XeTeX, which makes XeTeX can handle files that are encoded by other formats. But unfortunately, XeTeX provides no interface to specify the output encoding format, so that it would write files with UTF-8 in every situation, even if we put \XeTeXinputencoding or \XeTeXdefaultencoding at the beginning of the master .tex file.

A MWE that shows this problem:

\XeTeXdefaultencoding "GBK"
\documentclass{article}
\begin{document}
\tableofcontents
\section{测试一}

\clearpage
\section{测试二}

\clearpage
\section{测试三}

\clearpage
\section{测试四}

\clearpage
\end{document}


(Note that there is no selection of CJK fonts here.)

If you save this piece of code as GBK (M\$ calls it cp936), and compile it by XeLaTeX, no error will be reported. However, you would not be surprised that the .toc and .aux files are written as UTF-8.

The fact interpreted above will lead to another embarrassing tangible. These temp files were written by UTF-8, whereas the master file was saved as GBK. Hence, if we put \XeTeXinputencoding "GBK" at the begining of the master file, temp files will be read as UTF-8; if we use \XeTeXdefaultencoding "GBK", then temp fils will be read as GBK (but they themselves are UTF-8).

ERROR occurred so quietly but sadly.

Is there any clues or hints?

• Isn't using UTF-8 alone better? – egreg Jan 13 '14 at 13:33
• @egreg Yes indeed. But there are many old Chinese .tex documents were saved in GBK (China mainland, Big5 in Taiwan) using pdfLaTeX + CJK. We hope that we could find a comfort way to help these documents to migrate smoothly to XeLaTeX with xeCJK. – Ch'en Meng Jan 13 '14 at 13:38
• @egreg What's more, these users are not able to understand the difference between these two encoding formats. Forcing them changing the encoding is not a easy-achieving scheme. – Ch'en Meng Jan 13 '14 at 13:41
• iconv -f GBK -t UTF-8 filename.tex > filenameUTF8.tex is the best procedure, I believe. There are also GUI converters. – egreg Jan 13 '14 at 13:55
• 我不认为这个功能太值得做，旧文档没必要迁移，没必要用 XeTeX。新文档更不需要迁移这个功能，说服吴凌云打包时保持新版本 WinEdt 用 UTF-8 编码而不是 GBK 编码会更有意义。 – Leo Liu Jan 14 '14 at 4:22

If you can't convert to utf8 (which is naturally the best as it will make life much easier) imho the best is to add \XeTeXinputencoding "GBK" at the begin of every file (master + input files) which uses this encoding.
\XeTeXdefaultencoding is simply misnamed: it can't be used to declare the default encoding of a complete project. (But you can use it to declare the encoding of a bundle of files - if you are sure that auxiliary files don't interfere).
• UTF-8 was designed to make life much easier and better, however, there is an precondition, that everyone do use this encoding format. Tens of millions of old .tex documents were written and saved as GBK, and thus the precondition does not hold. Hence, finding a way to help these user (have old .tex documents saved as GBK) moving from (pdf)LaTeX + CJK to XeLaTeX + xeCJK really needs to be addressed. – Ch'en Meng Jan 14 '14 at 2:45