# Unicode characters changing order in listings (XeTeX)

I am trying to input code listings with unicode characters and everything seems to work (more or less) except some unicode characters being displayed out of order. For example instead of

służąca do obsługi

I get

łżąsuca do łobsugi

How can I fix that?

EDIT: I was asked for providing some code, but unfortunately I doubt it will be any helpful. Basically code like this:

\documentclass[11pt,polish]{article}

\usepackage{fontspec}
\usepackage{listings}
\usepackage{xunicode}
\usepackage{xltxtra}
\usepackage{babel}

\setmainfont{Charis SIL} % Some font with good unicode support

\begin{document}
\begin{lstlisting}
służący
\end{lstlisting}
\end{document}


will output this:

łżą

sucy

(yes there is a line break, I don't know where it came from, when I input code in form of a file there are characters out of order, but no line breaks).

-
I'm guessing the problem is that characters that don't fit into one byte are moved to the beginning of the word for some reason... – mrpyo Nov 6 '12 at 19:01
The display problem occurs in LyX or the final pdf or both? – scottkosty Nov 7 '12 at 0:10
@scottsky Final pdf. It's listings package problem like others have explained. LyX uses native engines for display in editor, not LaTeX. – mrpyo Nov 7 '12 at 11:09

Package listings is not aware of characters with character code greater than 8-bit. Internally the package redefines each character as active character. The following example allows the support of additional Unicode characters by adding them to the list of extended chars.

Supporting all Unicode characters this way is difficult:

• High memory usage.
• There are different classes of characters (\lst@ProcessLetter, \lst@ProcessOther, …)

Example file:

\documentclass[11pt,polish]{article}

\usepackage{fontspec}
\usepackage{listings}
\lstset{extendedchars}

\begingroup
\catcode0=12 %
\makeatletter
\lst@CCECUse\lst@ProcessLetter
łżą% *** add Unicode characters ***
^^00% end marker
}%
\endgroup

\begin{document}
\begin{lstlisting}
służący
\end{lstlisting}
\end{document}


-
Btw. Why no one have updated listings package to support unicode? I mean TeX community seems to be quite active, and it doesn't look to me as something very hard to fix... – mrpyo Nov 7 '12 at 12:14
@mrpyo It is very hard to fix. For one thing, Unicode can appear in different flavors, multiple byte strings or as big chars (XeTeX/LuaTeX). For the other thing, the internal parsing stuff very likely needs to be rewritten. – Heiko Oberdiek Nov 7 '12 at 12:50
@HeikoOberdiek, is this the fix to listings, ctan.org/pkg/listingsutf8? or is it something unrelated? (I just found it be chance) – alfC Nov 23 '14 at 18:55
@alfC It's a limited workaround. The listings has to be in a file, the environment and inline form are not supported. Also an 8-bit encoding is needed, which contains all the needed characters. – Heiko Oberdiek Nov 23 '14 at 19:04
@HeikoOberdiek, Under which conditions "an one-byte encoding exists that the file can be converted to" (quote from the manual)? – alfC Nov 23 '14 at 19:07

Here's another solution, if listings isn't a requirement. My PythonTeX package provides syntax highlighting for most languages, via the Python highlighting library Pygments. The package offers full Unicode support, under pdflatex, lualatex, and xelatex. It works under Windows, Linux, and OS X.

PythonTeX provides a pygments environment for typesetting blocks of code, a \pygment command for inline use, and an \inputpygments command for bringing in external files. All of these take optional fancyvrb commands, so you can easily get features like line numbering.

Here's a slight modification of your minimal example:

\documentclass[11pt,polish]{article}

\usepackage{fontspec}
\usepackage{pythontex}
\usepackage{xunicode}
\usepackage{xltxtra}
\usepackage{babel}

\setmainfont{Charis SIL} % Some font with good unicode support

\begin{document}

% Just change text' to the correct language for your code
\begin{pygments}{text}
służący
służąca do obsługi
łżąsuca do łobsugi
\end{pygments}

\end{document}


If your source code uses any tabs, you should use xelatex with the -8bit` option, so that tabs can be correctly written to file by xelatex (see this for details).

-