# Write18: LuaTeX has an encoding problem, where XeLaTeX does not

xelatex (with option --shell-escape) compiles this document fine:

\documentclass{article}
\usepackage{fontspec}
\begin{document}
LuaTeX is not
\immediate\write18{echo uber | sed y/u/ü/ > inc.tex}
% or: \directlua{os.execute("echo uber | sed y/u/ü/ > inc.tex")}
\input{inc.tex}.
\end{document}


lualatex (with --shell-escape) returns this error:

sed: -e expression #1, char 7: strings for y' command are different lengths


Obviously LuaLaTeX has an encoding problem and treats “ü” as two characters, where XeLaTeX gets things fine. Is there any workaround?

• In that particular case, I would try replacing sed y/u/ü > inc.tex by sed s/u/ü/g > inc.tex. – Édouard Jul 7 '13 at 22:21
• @Édouard Of course. I just gave a MWE, but in the real world the sed script is much more complex. – BertS Jul 8 '13 at 6:48

LuaTeX resets the locale settings to C, the documentation says at the end of section "3.2 LUA changes":

In stock LUA, many things depend on the current locale. In LUATEX, we can’t do that, because it makes documents unportable. While LUATEX is running if forces the following locale settings:

LC_CTYPE=C
LC_COLLATE=C
LC_NUMERIC=C


Then sed does not see an Unicode character, encoded as UTF-8, but two bytes.

In case of sed the setting LC_CTYPE=en_US.UTF-8 helps:

\documentclass{article}
\usepackage{fontspec}
\begin{document}
LuaTeX is not
\immediate\write18{echo uber | env LC_CTYPE=en_US.UTF-8 sed y/u/ü/ > inc.tex}
\input{inc.tex}.
\end{document}

• Does the change in settings revert to the original setting once the sed command is done, or is there a risk to change the behaviour of LuaTeX for the rest of the document? – Bruno Le Floch Jul 7 '13 at 22:24
• I would guess that interpretation of the string is left entirely to the shell. The modified environment thus affects only the call to sed in this particular case. This can be checked by replacing the command by echo uber | env LC_CTYPE=en_US.UTF-8 sed y/u/ü/ > inc.tex | locale. – Édouard Jul 7 '13 at 22:40
• @BrunoLeFloch: The environment variable is set via the program env that calls sed with the changed environment. Thus the setting is only valid for the called sed and its subprocesses. And without env` the setting is only valid for the sub shell, which LuaTeX calls for the system command, and its subprocesses. But the environment of the parent process (LuaTeX) is not changed at all. – Heiko Oberdiek Jul 7 '13 at 22:41