# LaTeX baffled by BOM---Unicode's byte order mark.

Unicode applies the convention of using a byte order mark as signature at the beginning of a text stream, identifying the encoding used within it. The following three bytes at the beginning of a file: EF BB BF, identifies this file as a UTF8 file. Vi and most text editors gracefully ignore this signature. Open office does that as well, but it adds this signature at the beginning.

Now, if I open a TeX file with OpenOffic.org, and I do that quite a lot (for mixed directionality editing), the signature is added, and baffles LaTeX, which produces an error message such as:

 ! LaTeX Error: Missing \begin{document}.

See the LaTeX manual or LaTeX Companion for explanation.
Type  H <return>  for immediate help.
...

l.1
��\documentclass{article}


whereas a simple dump of the file does not show the problem. Is there a way to eliminate this problem, while staying in the realm of latex?

-
The usage of a script to remove and later re-add the BOM might be your best option. –  Martin Scharrer Feb 7 '11 at 13:43
Personally, I think the problem here is with OpenOffice. I would try to avoid using it to edit .tex files. Doesn't LyX have "track changes" features that you could use instead? –  Seamus Feb 7 '11 at 14:01
LyX is probably a good option for many; but, it did not work for me. The "track changes" is not what I called OpenOffice to do, but rather mixed bidi editing. –  Yossi Gil Feb 7 '11 at 16:39
Related question on StackOverflow: How do I remove the BOM character from my xml file –  Martin Scharrer Oct 4 '11 at 8:39
I think this is quite a bothersome issue, and Ulrike's workaround is not sufficient. Would it not be reasonable for the TeX engines to ignore the BOM? –  einpoklum Dec 16 '11 at 17:04

The inputenc package can only work from when it is loaded, so obviously not at the very start of the document. Until then, only ASCII should be used.

For real unicode handling, use a Unicode capable engine like XeTeX or LuaTeX (in your case, xelatex or lualatex). Then you don't need inputenc (and likely want to change some other packages, too).

-

You can e.g. change the \catcode of the three problematic bytes before you input your file:

pdflatex \catcode239=9 \catcode 187=9 \catcode 191=9 \input test-bom


You should then reset the catcodes in the document to 12.

-

The non-Unicode engines don't know anything about Unicode or UTF-8. A similar problem would arise if you tried to place a BOM at the beginning of a Unix script, which would have the kernel ignore the shebang line.

Avoid BOMs in TeX or LaTeX documents, they are neither necessary nor recommended. The Unicode-capable engines XeTeX and LuaTeX, however, handle BOMs just fine.

-
This is indeed a challenge, but how about telling latex, through the command line, or by some other clever method, to load the package prior to loading the main file. The BOM is necessary if you want to edit mixed directionality text, since OpenOffice is the main solid editor that does that best, enabling mixed directions in individual lines. –  Yossi Gil Feb 7 '11 at 13:25
@Yossi: See Ulrike's answer. –  Philipp Feb 7 '11 at 14:00
You may try to do latex '\RequirePackage[utf8]{inputenc}\input{file}'. –  Paŭlo Ebermann Feb 7 '11 at 14:51
@Yossi : for mixed directionality text see XeTeX or LuaTeX –  Matsaya Feb 7 '11 at 17:08