What is the status of generating LaTeX from handwriting (i.e., OCR)?

Question

I type fast enough that for most things it's not a pain, but I have a few big stacks of old course notes I'd like in LaTeX which I'm dreading having to go through. So, I'm just wondering what the best solution for handwriting -> LaTeX is so far, if any.

Answer 1

So, I'm just wondering what the best solution for handwriting -> LaTeX is so far, if any.

There is none, and if there’ll ever be one it’s probably years, if not decades off. I know people who are currently working on recognizing just the layout of a document, i.e. recognizing that a paper represents a letter, etc.

That works fairly well, but it’s still research level, and going from recognizing the layout to replicating the layout using LaTeX is a big, non-obvious step. And we’re not even talking about text recognition itself.

Just text recognition (i.e. ignoring any layout issue) works fairly well today but only for plain text, not with any formatting.

That said, there’s JMathNotes (scroll almost to the bottom) which recognizes basic formulas and produces LaTeX output. It’s a nice and quite powerful proof of concept.

But it’s important to realize that even though many of the individual building blocks exist, piecing together a working solution is hard.

Answer 2

I have summarized the most of the current answers here or below.

I will cover now some papers, work in progress. I understand it so that the Tapio -paper, before preprocessing, uses LP -methods for his formulated QP -puzzle. The Knerr -paper uses discretization of words so one word can have many routes, now getting easily an exponential network-optimization problem. The ON-REC -method is almost the same as the REC-REC -method but some modifications. Knerr has publized a new paper "Combining diverse systems for handwritten text line recognition" (2011). The Japananese paper contains pretty much no details, mostly programming-biased rhetory or worse marketing of their InftyReader.

Academia

I. Ernesto Tapia from Freie Uni Berlin, something here but many pages broken, has publications here and his mostly-cited paper below.

• "Recognition of on-line handwritten mathematical formulas in the e-chalk system" here

Key terms: empirical risk, structural risk, pattern recognition, QP -problem, Lagrange multipliers, theory developed by Vapnik and Chervonenkis (VC),

Perhaps important terms: radial basis functions (RBFs), polynomial kernels, hyperbolic kernels, sequential minimal optimization (SMO), --

II. Stefan Knerr (CEO of Vision Objects here, over 70 employees) has publications here, they approach the problem differently -- firstly quantifying different segments into markov chains. Then they get some sort of network -optimization problem that I cannot yet fully understand but trying.

• "Recognition-directed recovering of temporal information from handwriting images" -paper converts words into finite state-machines like the picture here.

Key terms: frame-extraction/vector-quantization/discreate-HMMs here, discrete Hidden Markov Models (HMMs), Tabou method (1984), Baum–Welch training algorithm, ON-REC system, REC–REC system,

"(i) a left–right scan of the word—referred as SCAN–REC further, (ii) a time order of the strokes recovered previously from the static image—referred latter as REC–REC, (iii) a time order of the strokes corresponding to the true online ordering—referred as ON–REC." (the Knerr -paper)

Perhaps important things: IRONOFF database,

III. Japanese researchers such as Masakazu Suzuki, Toshihiro Kanahori, Nobuyuki Ohtake and Katsuhito Yamaguchi -- apparently something to do with Ideal Group -companies such as InftyReader here. Anyway, their most-cited paper below shows a more programming-biased -prototype.

• "An Integrated OCR Software for Mathematical Documents and Its Output with Accessibility" (2004)

Perhaps Key terms: Unified Braille Code (UBC) by BANA (Braille Authority of North American), working requires "scanned binary images in either 600 DPI or 400 DPI"

Puzzles

1. Is this quadratic programming problem (QP) image here? Source is the Tapia. I understand this so that the author linearize the quadratic programming problem here with Lagrange -multiplier method.

2. "baseline structure analysis method developed by Zanibbi et al [14]. The idea is that mathematical notation can be described as a hierarchical structure of nested baselines." (the Tapia -paper I added the bolding)

3. The Knerr -paper mentions "the second optimization process uses directed graph models" and "The number of possible paths of the ‘‘REC– REC’’ approach for a word with N segments is 2N!" (I added bolding)

Future development

• Open-source OCR system for FPGA?. Glen recommends to look for "dynamic programming algorithms and systolic array processors" here: I think the key is to break the problem into things like subgame perfect equilibriums so it can be parallerized and done fast.

• Real-time skeletonization using FPGA, A real-time matching system for large fingerprint databases (almost the speed of ASIC) -- it is non-trivial task to do the cover and skeletonization.

Products

• Microtask breaks the OCR-detection into games where players identify parts that are too hard for computers to detect. In exchange, the gamers can receive digital currencies.

Answer 3

How about reading your notes and use speech recognition ? That would be an alternative to OCR.

Answer 4

An iteresting research on Mathematical Information Processing is explained here at the InftyReader project page. It's a Japanese research group.

Answer 5

I'm the developer of VisionObjects, but I'm not in the research team.

TeX characters are not all supported. Gregory posted on HN a list of chars we support.

Suggestions are welcomed. Do not hesitate to send us missing symbols or UI improvement ideas.

Answer 6

Inlage (http://www.inlage.com) is a Latex editor which offers recognition of handwritten formulas on Windows 7. It makes use of the Windows 7 math input panel and converts the generated MathML to Latex. See a video of how it works at Inlage II feature: Math Input Panel to LaTeX.

Note: I'm in no way affiliated with this program. TexTablet might be a free alternative.

Answer 7

Very impressed by VisualObjects Web Equation

Screenshot _{(of doncherry's clueless scribbling)}:

Screenshot _{(of Aymon's expert scribing)}:

Answer 8

Given that it hasn't been mention, detexify basically takes handwritten text and produces TeX/LaTeX code (granted on a single symbol scale).

Answer 9

It's not a LaTeX solution, but very usefull to me: Get a new version of a speech recognition programm and read aloud to your computer.

This is a lot faster than typing, even if you were a professionell typewriter. I bought a "premium" version. There you can define your own speech commands. So the command "techenumeration" makes the software type

\begin{enumerate}
\item
\end{enumerate}

Give it a try, the software works way better than some years ago.