Take the 2-minute tour ×
TeX - LaTeX Stack Exchange is a question and answer site for users of TeX, LaTeX, ConTeXt, and related typesetting systems. It's 100% free, no registration required.

I'm thinking of making a simple script to turn Project Gutenburg texts into LaTeX documents, so that they are nicely typeset and ready for reading. However to do that I'll need a fairly good list of what characters I should be handling specially.

My plan is to run a sed script on it, making substitutions, then add a basic preamble at the start, end document at the end, then run LaTeX on it. It doesn't seem too hard, but given the length of most of these ebooks I'd rather not try and find all the things my script misses by hand.

Here is the list I have so far:

  • If I use UTF8 I should already be able to use most fancy characters without adjustment.
  • "foo" for ``foo''
  • 'foo' with `foo'
  • Replacing ... with \ldots
  • Replacing hyphens with --- when appropriate (that could be a challenge)
  • ~ with $\sim$

What else am I missing?

I know using the HTML versions would look a lot nicer, but the eBooks have lots of fancy stuff that looks really hard to convert.

Also, does anyone know much about GutenMark? It seems to do the same thing as I just proposed.

share|improve this question
2  
1  
Perhaps, if they include titles, add a control space. For example, Mr. Drofnats to Mr.\ Drofnats, or better Mr.~Drofnats. –  Werner Dec 19 '11 at 18:22
    
I don't suppose you could place those in answers to make them easier to keep track of? –  Canageek Dec 19 '11 at 18:25
    
For quotation marks, if they are already "smart" quotes, then using UTF8 (especially XeLaTeX or LuaLaTeX) will work. If they are plain old " then you can fix this with the csquotes package. See fontspec with Helvetica breaks quotes for an example. –  Alan Munn Dec 19 '11 at 18:44
1  
Relevant Rant? –  Aditya Dec 19 '11 at 19:08
show 1 more comment

2 Answers

up vote 12 down vote accepted

If you are going to spend any serious time on it, I recommend you parse the HTML version rather than the text version first through a pre-processor. I would rather use Perl, Lisp, Python or Ruby rather than sed.

The reasoning to parse the HTML rather the text file, is that the HTML is already structured as well as problematic characters probably already escaped and hence easier to parse and translate. My choice of the computer languages above, is that you can find libraries for html parsing that you can modify.

Before you do any pre-processing you need to pick up the encoding of the files. Gutenburg files come in many flavours.

For LaTeX, I would certainly use utf8 and preferably XeLaTeX. The problematic characters that need to be escaped are well known and are listed in the TeXbook, characters such as the underscore, hash signs and the like. Get prepared for a lot of these � on the utf8 side.

The challenges though are mostly on the parsing side. For example the Gutenburg ASCII files will show the page number as just a number. You will need to catch that and the many variants. Prepare for a joyful, frustrating but rewarding experience into natural language processing. IMHO the (La)TeX part is the easy part.

Added a small MWE to indicate some of the utf8 issues that you may encounter. Highly recommended reading before embarking on the Project is the fontencand inputenc manuals from where the example was developed.

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{trace}
\scrollmode % to run past errors check the log
\begin{document}
\obeylines
^^c3^^a4^^c3^^b6^^c3^^bc
These trigger errors
^^c3X ^^e1XY
%�
This \textcopyright\ will probably give you pdf 
issues  (try cut and paste in the pdf).
 \textcopyright
\end{document}
share|improve this answer
add comment

Consider including the following LaTeX corrections to HTML input:

  • If the source includes titles, add a control space. For example, Mr. Drofnats to Mr.\ Drofnats, or better Mr.~Drofnats.
  • Make sure you correctly format end-of-sentence abbreviations using \@.

Technically this will not "screw up LaTeX", but the LaTeX output should be typographically correct.

share|improve this answer
    
I am glad that you brought these two items up. In my answer I mentioned that the LaTeX part is the easy part. Parsing for name entities and sentence boundaries is part of the hard part. –  Yiannis Lazarides Dec 21 '11 at 5:15
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.