Take the 2-minute tour ×
TeX - LaTeX Stack Exchange is a question and answer site for users of TeX, LaTeX, ConTeXt, and related typesetting systems. It's 100% free, no registration required.

Unicode applies the convention of using a byte order mark as signature at the beginning of a text stream, identifying the encoding used within it. The following three bytes at the beginning of a file: EF BB BF, identifies this file as a UTF8 file. Vi and most text editors gracefully ignore this signature. Open office does that as well, but it adds this signature at the beginning.

Now, if I open a TeX file with OpenOffic.org, and I do that quite a lot (for mixed directionality editing), the signature is added, and baffles LaTeX, which produces an error message such as:

 ! LaTeX Error: Missing \begin{document}.

 See the LaTeX manual or LaTeX Companion for explanation.
 Type  H <return>  for immediate help.
 ...                                              

l.1 
 ��\documentclass{article}

whereas a simple dump of the file does not show the problem. Is there a way to eliminate this problem, while staying in the realm of latex?

share|improve this question
    
The usage of a script to remove and later re-add the BOM might be your best option. –  Martin Scharrer Feb 7 '11 at 13:43
    
Personally, I think the problem here is with OpenOffice. I would try to avoid using it to edit .tex files. Doesn't LyX have "track changes" features that you could use instead? –  Seamus Feb 7 '11 at 14:01
    
LyX is probably a good option for many; but, it did not work for me. The "track changes" is not what I called OpenOffice to do, but rather mixed bidi editing. –  Yossi Gil Feb 7 '11 at 16:39
2  
Related question on StackOverflow: How do I remove the BOM character from my xml file –  Martin Scharrer Oct 4 '11 at 8:39
    
I think this is quite a bothersome issue, and Ulrike's workaround is not sufficient. Would it not be reasonable for the TeX engines to ignore the BOM? –  einpoklum Dec 16 '11 at 17:04
show 1 more comment

3 Answers

up vote 12 down vote accepted

The inputenc package can only work from when it is loaded, so obviously not at the very start of the document. Until then, only ASCII should be used.

For real unicode handling, use a Unicode capable engine like XeTeX or LuaTeX (in your case, xelatex or lualatex). Then you don't need inputenc (and likely want to change some other packages, too).

share|improve this answer
add comment

You can e.g. change the \catcode of the three problematic bytes before you input your file:

pdflatex \catcode239=9 \catcode 187=9 \catcode 191=9 \input test-bom

You should then reset the catcodes in the document to 12.

share|improve this answer
add comment

The non-Unicode engines don't know anything about Unicode or UTF-8. A similar problem would arise if you tried to place a BOM at the beginning of a Unix script, which would have the kernel ignore the shebang line.

Avoid BOMs in TeX or LaTeX documents, they are neither necessary nor recommended. The Unicode-capable engines XeTeX and LuaTeX, however, handle BOMs just fine.

share|improve this answer
    
This is indeed a challenge, but how about telling latex, through the command line, or by some other clever method, to load the package prior to loading the main file. The BOM is necessary if you want to edit mixed directionality text, since OpenOffice is the main solid editor that does that best, enabling mixed directions in individual lines. –  Yossi Gil Feb 7 '11 at 13:25
    
@Yossi: See Ulrike's answer. –  Philipp Feb 7 '11 at 14:00
2  
You may try to do latex '\RequirePackage[utf8]{inputenc}\input{file}'. –  Paŭlo Ebermann Feb 7 '11 at 14:51
    
@Yossi : for mixed directionality text see XeTeX or LuaTeX –  Matsaya Feb 7 '11 at 17:08
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.