What characters in a normal text document will screw up LaTeX?

Question

I'm thinking of making a simple script to turn Project Gutenburg texts into LaTeX documents, so that they are nicely typeset and ready for reading. However to do that I'll need a fairly good list of what characters I should be handling specially.

My plan is to run a sed script on it, making substitutions, then add a basic preamble at the start, end document at the end, then run LaTeX on it. It doesn't seem too hard, but given the length of most of these ebooks I'd rather not try and find all the things my script misses by hand.

Here is the list I have so far:

• If I use UTF8 I should already be able to use most fancy characters without adjustment.
• "foo" for ``foo''
• 'foo' with `foo'
• Replacing ... with \ldots
• Replacing hyphens with --- when appropriate (that could be a challenge)
• ~ with $\sim$

What else am I missing?

I know using the HTML versions would look a lot nicer, but the eBooks have lots of fancy stuff that looks really hard to convert.

Also, does anyone know much about GutenMark? It seems to do the same thing as I just proposed.

Answer 1

If you are going to spend any serious time on it, I recommend you parse the HTML version rather than the text version first through a pre-processor. I would rather use Perl, Lisp, Python or Ruby rather than sed.

The reasoning to parse the HTML rather the text file, is that the HTML is already structured as well as problematic characters probably already escaped and hence easier to parse and translate. My choice of the computer languages above, is that you can find libraries for html parsing that you can modify.

Before you do any pre-processing you need to pick up the encoding of the files. Gutenburg files come in many flavours.

For LaTeX, I would certainly use utf8 and preferably XeLaTeX. The problematic characters that need to be escaped are well known and are listed in the TeXbook, characters such as the underscore, hash signs and the like. Get prepared for a lot of these � on the utf8 side.

The challenges though are mostly on the parsing side. For example the Gutenburg ASCII files will show the page number as just a number. You will need to catch that and the many variants. Prepare for a joyful, frustrating but rewarding experience into natural language processing. IMHO the (La)TeX part is the easy part.

Added a small MWE to indicate some of the utf8 issues that you may encounter. Highly recommended reading before embarking on the Project is the fontencand inputenc manuals from where the example was developed.

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{trace}
\scrollmode % to run past errors check the log
\begin{document}
\obeylines
^^c3^^a4^^c3^^b6^^c3^^bc
These trigger errors
^^c3X ^^e1XY
%�
This \textcopyright\ will probably give you pdf 
issues  (try cut and paste in the pdf).
 \textcopyright
\end{document}

Answer 2

Consider including the following LaTeX corrections to HTML input:

• If the source includes titles, add a control space. For example, Mr. Drofnats to Mr.\ Drofnats, or better Mr.~Drofnats.
• Make sure you correctly format end-of-sentence abbreviations using \@.

Technically this will not "screw up LaTeX", but the LaTeX output should be typographically correct.