Take the 2-minute tour ×
TeX - LaTeX Stack Exchange is a question and answer site for users of TeX, LaTeX, ConTeXt, and related typesetting systems. It's 100% free, no registration required.

Back in the old days, before I left the hallowed shores of my native country, I didn't much care what encoding my files were in. ASCII was enough for my great-great grandfather so it was enough for me. Now I live in the land of my great-great-...-great grandfathers who have a slightly mangled alphabet and ASCII no longer suffices. That would be fine if it were just me: I've learned to love UTF-8 and have even embraced xelatex. But sometimes I get sent documents in weird encodings, and sometimes I want to resurrect some old document, maybe to include a section in a new document, so from time to time I find myself wanting to change the encoding of a document.

So: how do I do that? (Included subquestion: how do I determine the encoding of a file?)

Notes:

  1. I realise that this is only tangentially related to TeX and friends, so am fully prepared to be told to look elsewhere, but I think that this is quite common and especially (due to the fact that it seems one has to be super-aware of encodings with TeX) important for TeXers.
  2. I don't have a specific example file in mind here, this is a "generic" question hoping to build a useful resource. So please answer in as full generality as you can and where you need to place restrictions, please make them clear. In particular, this will almost certainly have different answers depending on the OS.
  3. On the other hand, if you do know a super-snazzy-wizzy method that works just brilliantly when using Emacs at midnight with a full moon, then please do post it - just be sure to include whether you are assuming the strong lycanthropic principle or only the weak one.
  4. In light of those last two, I'd be happy for this to be CW with one answer gathering together all the techniques in a sensible grouping.
  5. If this question doesn't get closed and does work as I intend, these notes should probably be removed so as not to distract from the usefulness of the answers (and because we probably don't want TeX-SX to be the number one hit for the "strong lycanthropic principle").
share|improve this question
4  
@Andrew: Are you looking for a single-file or a 'job lot' approach? A number of editors can read and write different encodings, so for simple cases it's just a question of your editor. On the other hand, bulk changes sound like a question for SuperUser. –  Joseph Wright May 30 '11 at 11:37
    
@Joseph: Hadn't thought about that. The scenario I most encounter is "single file". I think that if I had a "job lot" then I'd figure it out once and for all and write a shell script to do it for me. But this happens just infrequently enough that I never remember from one occurrence to the next how to do it. (In case it's not clear, I'm also deliberately playing "new-users' advocate" in that I could probably figure it out for myself but I think that this would be a really useful thing to have on this site.) –  Loop Space May 30 '11 at 11:57
4  
I think that Joseph is right. Propably any decent editor has the capability to change the encoding of a file. Usually moving from ASCI to utf-8 is not a problem. The other way can lead to data loss (data corruption actually). Nowadays, I think that you should always work with Utf-8 regardless of the files contents. –  pmav99 May 30 '11 at 12:14
1  
@pmav99: I agree that one should, and I would tend to convert from "X" to UTF-8 whenever possible. Sometimes, though, I need to send files in particular encodings to people who are ... how shall I put it? ... less aware of encoding issues than myself. I also agree with the first sentence, but then apparently Emacs has the ability to change the universe so that my program is written already, but somehow M-x quantum butterfly doesn't work for me, so having actual instructions for the clueless (ie me) would (I think) be a useful resource for this site. –  Loop Space May 30 '11 at 12:34
    
I was going to mention file (which mostly works), but it failed to detect the encoding of a CP1251 text :-) For conversion, however, iconv is perfectly suited. –  Andrey Vihrov May 30 '11 at 13:04
show 1 more comment

8 Answers 8

Regarding Emacs: Sometimes I stumble upon a *tex-file encoded with latin-1 or latin-9, but the second line usually is \usepackage[latin1]{inputenc}. In Emacs, I delete this line and add via C-c C-m a new \usepackage, wait until Emacs finished looking which packages are installed and type inputenc. Emacs then suggests latin-1, I type utf-8 and Emacs asks, whether the whole buffer should be encoded with utf-8. YES! And Emacs recodes my file.

Otherwise look for the manpage of recode.

Alexander

share|improve this answer
    
Looks extremely useful! Are you using any particular Emacs library for TeX files? –  Loop Space May 30 '11 at 12:07
    
Uhm, you know what AUCTeX is? gnu.org/software/auctex You need that. A. –  Keks Dose May 30 '11 at 15:46
2  
Keks, that this is a facility of AUCTeX should be added to your answer. That's the sort of detail that's quite important. Not every user of Emacs uses AUCTeX (me, for example!). –  Loop Space May 30 '11 at 18:28
add comment

I recently have had some issues with the encodings of different files (*.bbl and *.tex for instance), resulting in missing glyphs or funny combinations in place of accented letters.

A solution that worked for me, in Linux (Ubuntu 10.04 distribution), was using the iconv program:

iconv -f source_encoding -t dest_encoding inputfile.tex > outputfile.tex
share|improve this answer
    
iconv is also available for Windows via MinGW –  matth Mar 1 '12 at 16:43
add comment

Changing the encoding of a file in emacs is easy, at least in GNU emacs at or beyond version 23, I think. First, visit the file in an emacs buffer. If emacs does not automatically visit the file with its current encoding, close the buffer, then type C-x C-m c to specify the file's current coding system, then open the file (using C-x C-f for example). Second, change the encoding by typing C-x C-m f to specify the new coding system. When you save the file, the new coding system is used.

The second issue is how to ensure emacs picks the right coding system next time you visit the file. For LaTeX files, it is usually not a problem, since latex mode looks at the \usepackage[…]{inputenc} to figure it out. There is a more general mechanism, but discussing that is outside the scope of this site, I think.

share|improve this answer
add comment

For single-file changes, I stick with my editor-of-choice (TeXworks), which can load and save a variety of encodings. The standard setting is UTF-8, but it will work with lots of others. (Auto-detection is based purely on the file itself, with no checking for inputenc or similar, so it may be that Emacs is a better choice in some cases.)

share|improve this answer
    
I think that inputenc is not a full proof way to detect the file's encoding. You can write whatever you want in the preamble and still save it in a different encoding. Then if you send it to someone else it will be decoded to gibberish. Propably a comment at the start of the file is the way to go (python like) –  pmav99 May 30 '11 at 13:17
add comment

Notepad++ is a very nice editor for Windows, it shows the current encoding and can convert the encoding to a different one. I use UTF-8 without BOM and the files worked well in all other editors afterwards. Notepad++ GUI for encoding conversion

share|improve this answer
add comment

iconv is also available for Mac OSX (besides Linux) although I don't know if it came with the OS or if I installed it with the huge MacTex-Works package.

I had to figure out to which encoding I accidentally saved my file. So I had to try out different encoding conversions.

In the terminal, this listed all the encodings available

iconv -l

This way I tried out one possible conversion after the other until my special characters (ü, ß, ...) were displayed correctly in the terminal

iconv -f MAC -t UTF-8 test.tex
iconv -f LATIN1 -t UTF-8 test.tex

This converted from UTF-8-MAC to UTF-8:

iconv -f UTF-8-MAC -t UTF-8 test.tex > test_UTF-8-MAC_UTF-8.tex
share|improve this answer
    
Useful to know. Checking my Mac system, I find iconv in /usr/bin/iconv. The date of creation matches that of other binaries that I would expect to be there by default (such as gzip and ssh) so I would say it is almost certain that iconv comes with the OS. –  Loop Space Aug 22 '12 at 8:12
add comment

There's also a GUI program called Charco that's available for Mac OS X, GNU/Linux and Windows

http://www.marblesoftware.com/Marble_Software/Charco.html

share|improve this answer
add comment

I can see that no-one mentioned the konwert utility (it is certainly present in ubuntu repositories). It has a lot of "filters" (for different encodings), can recognize some of them, nad has a lot of nice options, like removing trailing spaces, recoding cr/lf/crlf standards, escaping # $ % & \ ^ _ { | } ~ to TeX, rot13-ing the file and even more.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.