144

I type fast enough that for most things it's not a pain, but I have a few big stacks of old course notes I'd like in LaTeX which I'm dreading having to go through. So, I'm just wondering what the best solution for handwriting -> LaTeX is so far, if any.

13
  • 12
    OCR from handwriting is a hard problem by itself with mostly unsatisfactory solutions (AFAIK); I don't think adding equations into the mix will have any better solutions. Still, maybe someone knows better… Aug 10, 2010 at 0:52
  • 12
    Would be nice- but given that OCR even has trouble generating digital copies from scanned typeset text, this may be asking a lot.
    – Sharpie
    Aug 10, 2010 at 1:07
  • 1
    I think the best you can hope for from OCR is that you are able to run a full-text search on your scanned notes and have a good chance of finding the right one. This in itself is enough for me (I am not good about organizing notes into a way I can find them again) and the reason I plan to buy a tablet PC for my next laptop.
    – Lev Bishop
    Aug 10, 2010 at 1:37
  • 6
    This isn't really a question about LaTeX; the answers would not really be any different if Open Office's format was the target.
    – vanden
    Aug 10, 2010 at 6:30
  • 2
    How about reading your notes and using speech recognition ? That would be an alternative to OCR.
    – pluton
    Mar 16, 2012 at 23:55

16 Answers 16

72

So, I'm just wondering what the best solution for handwriting -> LaTeX is so far, if any.

There is none, and if there’ll ever be one it’s probably years, if not decades off. I know people who are currently working on recognizing just the layout of a document, i.e. recognizing that a paper represents a letter, etc.

That works fairly well, but it’s still research level, and going from recognizing the layout to replicating the layout using LaTeX is a big, non-obvious step. And we’re not even talking about text recognition itself.

Just text recognition (i.e. ignoring any layout issue) works fairly well today but only for plain text, not with any formatting.

That said, there’s JMathNotes which recognizes basic formulas and produces LaTeX output. It’s a nice and quite powerful proof of concept.

JMathNotes

But it’s important to realize that even though many of the individual building blocks exist, piecing together a working solution is hard.

5
  • JMathNotes looks like a very nice tool at first glance. I'll have to give it a try sometime.
    – Giel
    Aug 10, 2010 at 13:22
  • are there tools to convert formula in a pdf document to latex? besides JMathNotes Jan 5, 2012 at 6:25
  • 2
    Does this tool allow you to import an image containing an existing equation?
    – Werner
    May 14, 2015 at 19:51
  • 5
    Hey a decade has passed.
    – Rainb
    Jun 18, 2021 at 18:04
  • @Rainb Yes, and now a solution exists: mathpix.com/image-to-latex Feb 9 at 19:42
88

Very impressed by VisualObjects Web Equation

Screenshot (of doncherry's clueless scribbling):

Screenshot

Screenshot (of Aymon's expert scribing):

Screenshot

13
  • 9
    This tool is amazing!! Feb 2, 2012 at 6:53
  • 9
    Seriously, this is incredible.
    – Ryan Reich
    Feb 2, 2012 at 7:58
  • 4
    I shall now buy a graphics tablet. Impressive. Feb 2, 2012 at 8:28
  • 1
    @LordStryker its probably TOO perfect. Humans don't write like that. (If it is based on some sort of training). Just a thought ;) Feb 2, 2012 at 22:48
  • 3
    @AymonFournier You're right! This program actually promotes sloppy handwriting. All joking aside it is pretty swank. However it does need an 'erase' brush. If the interpreter messes up because of a slight mistake in input you have to start over from scratch (unless you're able to finesse it which did not work most of the time for me). Feb 2, 2012 at 23:29
46

The Mathpix app (for iOS only, Android coming soon) actually does this all on your phone via the camera.Mathpix demo

Just take pictures, and you can export as Latex, PDF, or you can get an Overleaf link (they have a really nice browser based editor). The iOS link is:

https://itunes.apple.com/us/app/mathpix/id1075870730?ls=1&mt=8

and the main website is just http://mathpix.com/.

Disclaimer: I'm the founder of Mathpix. I started working on this as a Stanford grad student in applied math, I hated how long it took to digitize my notes / homework sets. Anyway, Mathpix want to take the pain out of Latex for everyone, I hope this helps!

11
  • 1
    Nice! I just downloaded and tested your app. It works surprisingly well. I hope you get a chance to develop it further. There's definitely a lot of potential for increased productivity if this could be made to work reliably and also on more complex equations.
    – Janosh
    Feb 17, 2017 at 8:31
  • 3
    Yep working on it :) the last mile is the hardest Feb 17, 2017 at 9:58
  • Mathpix doesn't allow to import existing scanned pdfs though, right?
    – lucidbrot
    Jan 4, 2018 at 9:28
  • 2
    You can take screenshots of scanned pdf's using the Desktop app Apr 7, 2018 at 22:19
  • 3
    MathPix is awesome for single formulas and up to 5 lines (by my tests). They also provide an easy to implement API. I am also using their new desktop capturing tool for Windows. Good work!
    – Avatar
    Nov 1, 2018 at 11:55
24

I'm the developer of MyScript (formerly VisionObjects), but I'm not in the research team.

TeX characters are not all supported. Gregory posted on HN a list of chars we support.

Suggestions are welcomed. Do not hesitate to send us missing symbols or UI improvement ideas.

1
  • 7
    I'm not entirely clear what it is VisionObjects is providing - is it only the web apps?. I would dearly love a neat and accessible way to handwrite on my tablet and have the LaTeX inserted directly into (say) TeXnicCenter.
    – E.P.
    May 1, 2012 at 0:44
19

As the author of write-math.com, I think I can give this question an update.

First of all, there are two types of handwriting recognition: On-line and Off-line. On-line recognition means you can use the information how a symbol is written, whereas in off-line recognition you only have a pixel-map (aka "image"). Imagine on-line recognition as a movie where you get exact information where the tip of the pen was, whereas in off-line recognition you only get the end result. This means on-line recognition is simpler than off-line recognition as you can always just generate the end result.

I am doing research in on-line recognition.

There is an international conference on on-line handwriting recognition called ICDAR (international conference on document analysis and recognition) and a competition called CROHME. In this competition you get a very nice data (meaning: clearly written, no errors in the input as it often occurs in real live) and your classifier has to recognize the recording. The recordings are also very simple: The symbols are written on one line (no \begin{align}\end{align}, but multiple fractions are possible), a very simple set of 75 allowed symbols (0-9, a-e, i-k, n, x-z, A, B, C, X, Y - you can see that this list was designed to be minimal and not have difficult combinations like 0, O, o or \pi and \prod), no matrices, no delayed strokes (e.g. you write a < b and then decide to correct it to a \leq b). And still the best system in 2013 (by VisionObjects, see web demo) only got 60.36% correct.

There are three tasks which have to be solved:

  • single symbol recognition (quite easy): Given only a single handwritten symbol, find its LaTeX code
  • segmentation (MUCH harder): Given a handwritten equation, find which strokes belong to which symbol (not classifying the symbols but only saying "this is symbol a, this is another symbol b, ...")
  • structural analysis: given a list of symbols a and b, say if its ab or a^b or a_b. (I didn't try that by now, but I think that's relatively easy)

Why is segmentation so hard? It is the mind-blowing number of possibilities you have to segment. Suppose you have n=3 strokes. Then you could have the following segmentations:

  • 1: [[0, 1, 2]]
  • 2: [[0, 1], [2]]
  • 3: [[0, 2], [1]]
  • 4: [[0], [1, 2]]
  • 5: [[0], [1], [2]]

Possibility 3 is what makes it so complicated. I've collected a lot of recordings with write-math.com and manually segmented them. About 10% of all multi-symbol recordings have such delayed strokes (see above). The number of possibilities grows a shown in https://oeis.org/A000110 But even without delayed strokes, you still have 2^{n-1} possible ways to segment n stokes.

You can see my progress on this topic here: https://github.com/MartinThoma/hwrt/issues/21

All of my material (papers, presentations, tools) are here: http://martin-thoma.com/write-math/

TL;DR

If you have something non-trivial, you still have to write it yourself. But I try hard to change that :-)

17

It's not a LaTeX solution, but very useful to me: Get a new version of a speech recognition programm and read aloud to your computer.

This is a lot faster than typing, even if you were a professionell typewriter. I bought a "premium" version. There you can define your own speech commands. So the command "techenumeration" makes the software type

\begin{enumerate}
\item
\end{enumerate}

Give it a try, the software works way better than some years ago.

13
  • 2
    what is the name of the program?
    – qed
    Feb 9, 2014 at 22:29
  • 3
    @qed Dragon Naturally Speaking
    – Keks Dose
    Feb 10, 2014 at 7:58
  • Wouldn't keyboard shortcuts (which can be defined in most tex editors) always be faster than voice recognition? In any case, I really don't think this is an appropriate answer to the question since it doesn't address handwriting. Apr 5, 2016 at 22:57
  • @JessRiedel You got it the wrong way. To get a text into a file, keyboard shortcuts are not helpful, you have to type it, to dictate it or to scan a sheet of paper and rund OCR on the scan. And your opinion about an answer being »appropriate« is obviously not shared by many readers.
    – Keks Dose
    Apr 6, 2016 at 9:40
  • I think you may not understand what I suggested. The point is that for anyone who can type, it is faster and more reliable to type a keyboard shortcut that represents what you call "techenumeration" than it is to say that keyword out loud. This is why no one says "copy" and "past" to their computer; they use ctrl-c and ctrl-v. Apr 7, 2016 at 14:55
12

I have summarized the most of the current answers here or below.

I will cover now some papers, work in progress. I understand it so that the Tapio -paper, before preprocessing, uses LP -methods for his formulated QP -puzzle. The Knerr -paper uses discretization of words so one word can have many routes, now getting easily an exponential network-optimization problem. The ON-REC -method is almost the same as the REC-REC -method but some modifications. Knerr has published a new paper "Combining diverse systems for handwritten text line recognition" (2011). The Japananese paper contains pretty much no details, mostly programming-biased rhetoric or worse marketing of their InftyReader.

Academia

I. Ernesto Tapia from Freie Uni Berlin, something here but many pages broken, has publications here and his mostly-cited paper below.

  • "Recognition of on-line handwritten mathematical formulas in the e-chalk system" here

Key terms: empirical risk, structural risk, pattern recognition, QP -problem, Lagrange multipliers, theory developed by Vapnik and Chervonenkis (VC),

Perhaps important terms: radial basis functions (RBFs), polynomial kernels, hyperbolic kernels, sequential minimal optimization (SMO), --

II. Stefan Knerr (CEO of Vision Objects here, over 70 employees) has publications here, they approach the problem differently -- firstly quantifying different segments into Markov chains. Then they get some sort of network -optimization problem that I cannot yet fully understand but trying.

  • "Recognition-directed recovering of temporal information from handwriting images" -paper converts words into finite state-machines like the picture here.

Key terms: frame-extraction/vector-quantization/discreate-HMMs here, discrete Hidden Markov Models (HMMs), Tabou method (1984), Baum–Welch training algorithm, ON-REC system, REC–REC system,

"(i) a left–right scan of the word—referred as SCAN–REC further, (ii) a time order of the strokes recovered previously from the static image—referred latter as REC–REC, (iii) a time order of the strokes corresponding to the true online ordering—referred as ON–REC." (the Knerr -paper)

Perhaps important things: IRONOFF database,

III. Japanese researchers such as Masakazu Suzuki, Toshihiro Kanahori, Nobuyuki Ohtake and Katsuhito Yamaguchi -- apparently something to do with Ideal Group -companies such as InftyReader here. Anyway, their most-cited paper below shows a more programming-biased -prototype.

  • "An Integrated OCR Software for Mathematical Documents and Its Output with Accessibility" (2004)

Perhaps Key terms: Unified Braille Code (UBC) by BANA (Braille Authority of North American), working requires "scanned binary images in either 600 DPI or 400 DPI"

Puzzles

  1. Is this quadratic programming problem (QP) image here? Source is the Tapia. I understand this so that the author linearized the quadratic programming problem here with Lagrange -multiplier method.
  1. "baseline structure analysis method developed by Zanibbi et al [14]. The idea is that mathematical notation can be described as a hierarchical structure of nested baselines." (the Tapia -paper I added the bolding)
  1. The Knerr -paper mentions "the second optimization process uses directed graph models" and "The number of possible paths of the ‘‘REC– REC’’ approach for a word with N segments is 2N!" (I added bolding)

Future development

  • Open-source OCR system for FPGA?. Glen recommends to look for "dynamic programming algorithms and systolic array processors" here: I think the key is to break the problem into things like subgame perfect equilibriums so it can be parallelized and done fast.

Products

  • Microtask breaks the OCR-detection into games where players identify parts that are too hard for computers to detect. In exchange, the gamers can receive digital currencies.
2
11

Current answer: Smart Note (for iOs and Android). First "add equation", then "export:to LaTex".

Prior answer:

On the Samsung Galaxy Note, the proprietary program "S-Note" appears to have a licensed version of VisualObjects included, which it uses to do a very creditable job of converting handwriting to formatted equations. Downside: the LaTeX is not accessible (grrr). I actually was curious so I looked inside the application and decompiled some of the java source files... and I found that internally, it is using LaTeX! So honestly I'm sure that a creditable Java hacker (which I am not) could decompile the app, and with literally just a few lines of changes, make it convert to LaTeX rather than to a bitmap. If you grep the codebase for "EquationRecognition", you'll quickly find the relevant files. You could then recompile just those classes, re-bundle the app, and sign the hacked version. (Which of course would only be legal if you have a legit license to the app.)

I know, this is not actually a useful answer, but I just spend an hour or so finding this out, so I might as well share it.

3
  • +1 for the effort and because I've got a Galaxy Note.
    – McGafter
    Jun 12, 2015 at 8:34
  • I've now talked to both Samsung and MyScript (makers of the VisualObjects recognition code) about this. Samsung said "I'll tell the people in charge" and MyScript said: "...We also have some licensees who are working on applications that go beyond this feature set targeting academic / professional users of LaTeX and/or MathML, but those aren’t publicly announced yet."... In other words, the useful part of this answer is that you can either license their code, or wait for somebody else who did so to come up with a product, or read their papers and reverse engineer their code. Jun 18, 2015 at 19:25
  • Status update: MyScript's own Smart Note app on Android now has both "add equation" and "export to LaTeX" functions, so this works for me. Jul 10, 2016 at 14:55
9

This solution may not answer your question entirely, but if just want a write-on-the-fly solution on Windows with equations as well as handdrawing pictures, then try using TeXStudio (http://www.texstudio.org/). This software has integrated the Windows write recognization engine into itself after my request a few years ago on the software's open-source development issue tracker. See the screen snapshoot enter image description here,

the write LaTeX option is under the Wizards menu named as Math Assistant. It will then open up an OCR window as shown in the figure below.

enter image description here

Handwrite your equations on the panel, and then you only need to copy the equations shown on top as a LaTeX code to your TeXStudio input window. Or, you can just press the insert button to insert the tex code to your main text. Errors can be fixed by rewriting your equations or just by correcting the generated LaTeX code.

I have used this software to take notes on Physics classes where math and graphics are needed all the time. I use the Insert Graphic tool under the Wizards menu of the software to insert hand-drawing pictures which can be done easily from OneNote on the fly if you have a tablet (I use Thinkpad X200 tablet, Thinkpad Tablet 2 and Thinkpad Yogo 460 for taking notes electrically). If you really want to handwrite everything, you can use the default writing panel from Windows (Windows 7 and above I guess) to input your text as well, and then use the two tools I have introduced above to input handdrawing diagrams and equations all into LaTeX format.

One real example of compiled output using this technique and this set of tools can be found in my shared Notes on Classical Mechanics on Github.

8

An iteresting research on Mathematical Information Processing is explained here at the InftyReader project page. It's a Japanese research group.

6

Given that it hasn't been mention, detexify basically takes handwritten text and produces TeX/LaTeX code (granted on a single symbol scale).

5
  • 4
    I don't think this is a valid answer to this question. While it would be entirely unpractical if it worked, it doesn't even work: I attempted drawing an e and an E, and Detexify didn't recognize them, obviously, because it's only designed for recognizing symbols that need to be input via a special command, not by a single character.
    – doncherry
    Feb 2, 2012 at 13:48
  • 1
    @doncherry I agree that detexify has its limitations, but I think it represents "the status of generating LaTeX from handwriting." It is the best solution that I know of. I like it better than the web equation since it gives you its best guesses.
    – StrongBad
    Feb 2, 2012 at 14:06
  • This is indeed the state of the art for single characters, while Aymon's answer gives the state of the art for formulas. Feb 2, 2012 at 14:15
  • 3
    Detexify is not made for handwriting as input, but for a mouse-scribbled symbol that should look as similar to the printed symbol as possible. This is decisively different from handwriting recognition. Furthermore, Detexify isn't made for recognizing more than one symbol at once and most likely never will be, unless its purpose is changed. Saying it is "the status of generating LaTeX from handwriting" is like saying "cars are the status of flying submarines" -- they really suck at that and they'll never be good at it, unless their purpose undergoes a serious change.
    – doncherry
    Feb 2, 2012 at 14:22
  • 1
    @BrunoLeFloch: This is where I like to differentiate between characters and symbols: Roughly saying, the things on your keyboard are characters (unless you're using Neo) -- these usually can be input directly in LaTeX and they won't be recognized by Detexify; it doesn't recognize any normal Latin letters. Other glyphs that actually represent something are symbols. $ means dollar, means infinity -- but w doesn't mean anything.
    – doncherry
    Feb 2, 2012 at 14:26
6

Inlage (http://www.inlage.com) is a Latex editor which offers recognition of handwritten formulas on Windows 7. It makes use of the Windows 7 math input panel and converts the generated MathML to Latex. See a video of how it works at Inlage II feature: Math Input Panel to LaTeX.

Note: I'm in no way affiliated with this program. TexTablet might be a free alternative.

1
  • The Math Input Panel works really great! It is part of default Windows 7, in order to start it, just open the start menu and type mip.exe. Also note the related question: tex.stackexchange.com/questions/3755/…
    – matth
    May 22, 2012 at 13:35
5

If you use LyX on Windows 7 or later, there is a Math Input Panel Helper.

It's a little program that converts the Math Panel output to LaTeX or MathML and lets you insert math directly to LyX (or any other LaTeX editor).

4

I have found another application named Equatio (Google Chrome extension for Modules and Documents) that have many functionalities. Here there is the lik of the site: https://www.texthelp.com/en-us/equatio-gratis-para-profesores/#. It is free for the teachers: https://www.texthelp.com/en-us/products/free-for-teachers/

enter image description here

I have seen that this program give the possibility to convert handwriting into LaTeX code. See the part of the comment in the site

EquatIO is much more than a replacement for pen and paper problem solving. It’s smart – predicting what expression you’re trying to write, transforming your handwriting into text, and ignoring those ‘umms’ and ‘errs’ when you dictate aloud.*

3

There is another important tool which also converts handwriting into LaTeX code by MyScript.

https://webdemo.myscript.com/views/math/index.html#

Here there is a proof make with my hand:

enter image description here

If you push in convert button you will have the conversion of your handwriting in newtxtext font.

enter image description here

2

The https://mathkey-app.com/ nowadays allows to use the https://www.myscript.com/ or the https://mathpix.com/ engine:

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .