# What characters in a normal text document will screw up LaTeX?

I'm thinking of making a simple script to turn Project Gutenburg texts into LaTeX documents, so that they are nicely typeset and ready for reading. However to do that I'll need a fairly good list of what characters I should be handling specially.

My plan is to run a sed script on it, making substitutions, then add a basic preamble at the start, end document at the end, then run LaTeX on it. It doesn't seem too hard, but given the length of most of these ebooks I'd rather not try and find all the things my script misses by hand.

Here is the list I have so far:

• If I use UTF8 I should already be able to use most fancy characters without adjustment.
• "foo" for foo''
• 'foo' with foo'
• Replacing ... with \ldots
• Replacing hyphens with --- when appropriate (that could be a challenge)
• ~ with $\sim$

What else am I missing?

I know using the HTML versions would look a lot nicer, but the eBooks have lots of fancy stuff that looks really hard to convert.

Also, does anyone know much about GutenMark? It seems to do the same thing as I just proposed.

-
– lockstep Dec 19 '11 at 18:19
Perhaps, if they include titles, add a control space. For example, Mr. Drofnats to Mr.\ Drofnats, or better Mr.~Drofnats. – Werner Dec 19 '11 at 18:22
I don't suppose you could place those in answers to make them easier to keep track of? – Canageek Dec 19 '11 at 18:25
For quotation marks, if they are already "smart" quotes, then using UTF8 (especially XeLaTeX or LuaLaTeX) will work. If they are plain old " then you can fix this with the csquotes package. See fontspec with Helvetica breaks quotes for an example. – Alan Munn Dec 19 '11 at 18:44
– Aditya Dec 19 '11 at 19:08

If you are going to spend any serious time on it, I recommend you parse the HTML version rather than the text version first through a pre-processor. I would rather use Perl, Lisp, Python or Ruby rather than sed.

The reasoning to parse the HTML rather the text file, is that the HTML is already structured as well as problematic characters probably already escaped and hence easier to parse and translate. My choice of the computer languages above, is that you can find libraries for html parsing that you can modify.

Before you do any pre-processing you need to pick up the encoding of the files. Gutenburg files come in many flavours.

For LaTeX, I would certainly use utf8 and preferably XeLaTeX. The problematic characters that need to be escaped are well known and are listed in the TeXbook, characters such as the underscore, hash signs and the like. Get prepared for a lot of these � on the utf8 side.

The challenges though are mostly on the parsing side. For example the Gutenburg ASCII files will show the page number as just a number. You will need to catch that and the many variants. Prepare for a joyful, frustrating but rewarding experience into natural language processing. IMHO the (La)TeX part is the easy part.

Added a small MWE to indicate some of the utf8 issues that you may encounter. Highly recommended reading before embarking on the Project is the fontencand inputenc manuals from where the example was developed.

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{trace}
\scrollmode % to run past errors check the log
\begin{document}
\obeylines
^^c3^^a4^^c3^^b6^^c3^^bc
These trigger errors
^^c3X ^^e1XY
%�
This \textcopyright\ will probably give you pdf
issues  (try cut and paste in the pdf).

• If the source includes titles, add a control space. For example, Mr. Drofnats to Mr.\ Drofnats, or better Mr.~Drofnats.
• Make sure you correctly format end-of-sentence abbreviations using \@`.