# Converting Markdown to LaTeX, in LaTeX

I've started to use markdown for my notes simply because its faster than writing latex, but I would prefer to have the extra power to LaTeX. Is there/would it be possible to have a markdown environment which converted the markdown to latex? Is LaTeX powerful enough to parse another language?

Side question: Is LaTeX Turing complete?

I guess one option would be to pass the data to a script I wrote to convert them. I have read this is possible, is it possible to have latex pipe data into the stdin for another program?

-
latex can parse xml (see xmltex) and parse tikz syntax (see any question on this site:-) so it could probably parse markdown someone may well have done it, not sure though (hence this is a comment rather than answer) side answer: yes –  David Carlisle Mar 9 '13 at 23:54
Hmmm... lualatex, hmmm... lunamark... Mumble, mumble... –  JLDiaz Mar 9 '13 at 23:54
That looks great. Is there a way to set it up that I could have both LaTeX and markdown in the same file? It might get a bit messy if I didn't do that. –  superbriggs Mar 9 '13 at 23:55
Why not just write markdown, put the latex code inside and run the file through pandoc. As the last, finishing step, you may tweak pandocs latex-output and compile it with latex, if you are not able to get the preferred output directly from pandoc. –  Sveinung Mar 10 '13 at 9:40
TeX is Turing-complete, but it is slow and painful to use for many pure-programming tasks. If I understand correctly, this is one of the reasons why luatex was invented. It is somehow similar to the relation between pdf and javascript: pdf derives from ps, which is a Turing-complete language in theory, but they added Javascript on top of it anyways. –  Federico Poloni Mar 10 '13 at 10:29

A proper markdown parser is a task too complex for latex. Not because TeX is not a Turing complete language (it is), but because it would be very difficult to implement, and probably will have a very poor performance.

One idea which immediately comes to mind is to use LuaTeX, and code the markdown parser in Lua language. This sound certainly feasible.

According with Wikipedia, there are only three Markdown parsers implemented in Lua:

• markdown.lua. It is simple, and it would be probably easy to integrate into a LuaLatex package, but currently it only outputs HTML, so a complete rewrite of the output part should be done to generate latex code instead.

• lunamark. This is very complete. It can output a variety of formats, including LaTeX, and supports a lot of non-standard extra modules. However, it has a number of dependencies of other Lua packages, so in order to integrate it with LuaTex it should be stripped down (for example, template support could be removed, since the template will be the document which uses it, and also all output formats except LaTeX).

• lua-discount This is a lua binding to a parser written in C, so it can be discarded because that would not be embeddable in LuaTeX.

So the Lua approach basically reduces to one of those two:

1. Change markdown.lua to make it output LaTeX
2. Strip down lunamark to remove all unnecessary stuff.

I'm not sure which one will be easier. Probably 1 will produce some useable in a shorter time, but 2 looks like a better long term solution.

Also, the interface between LaTeX and Lua should be defined. A sensible approach would be:

% Latex stuff....
\begin{markdown}
# First section
etc..
\end{markdown}


However I'm very new to the LuaTeX world, and (currently) I don't know how to pass to a Lua function all the text inside the markdown environment. Also we should prevent TeX to tokenise all that text.

A possible idea is to use some verbatim-like tricks to write the contents of the environment to a file, and then use lua to process that file and insert back the resulting tex in the main document (a tex.sprint() would do, wouldn't?)

I would like to hear the opinion of the LuaTeX experts here...

PS: Does this count as an answer? Or should I delete it and re-post as a question instead?

-
Did you see this: tex.stackexchange.com/questions/38150/… ? –  mbork Mar 10 '13 at 16:54
@mbork No, I didn't know about it. I bookmarked it :-) Thanks! But as you can see in my other answer, I managed to do it via auxiliar files. –  JLDiaz Mar 10 '13 at 17:10
Yes, but aux files are much worse, especially in terms of speed. –  mbork Mar 10 '13 at 17:16
@mbork I coded a new solution based in the answer you linked. See my updated answer. I cannot detect any speed improvement, but I guess it would be noticeable in bigger documents. –  JLDiaz Mar 10 '13 at 17:42
Thanks! Now imagine a file with, say, 500 Markdown snippets interspersed with normal LaTeX code. (Not that difficult to happen, say, in a Markdown user's guide, for instance.) Overhead for opening, writing to and closing so many files would be considerable. –  mbork Mar 10 '13 at 18:00

Here's Pandoc-based solution. You will have to enable --shell-escape for this to work, since it uses \write18. Depending on what you want, you may need to customize the Pandoc options.

\documentclass{article}

\usepackage{fancyvrb}

\newenvironment{markdown}%
{\VerbatimEnvironment\begin{VerbatimOut}{tmp.markdown}}%
{\end{VerbatimOut}%
\immediate\write18{pandoc tmp.markdown -t latex -o tmp.tex}%
\input{tmp.tex}}

\begin{document}

\begin{markdown}
# Section

Some text that goes on for a while.

A list:

* Item
* Another item

\end{markdown}

\end{document}

-
I've been using this with good results (some issues, but really nothing too major). Any suggestions as to how I can incorporate comments inside the markdown environment, so that they are not displayed in the output? –  a different ben Aug 14 '13 at 1:29
@adifferentben Normal TeX comments don't work, because a % in markdown gets turned into a literal character \%. You could redefine \% within the markdown environment, but then that would be a problem if you ever wanted a literal %. Probably the simplest option is just to define a comment macro, something like \newcommand{\comment}[1]{}. This will pass right through the conversion process without being altered, since Pandoc doesn't do anything to content it identifies as being LaTeX. –  G. Poore Aug 14 '13 at 2:34
@adifferentben you can use HTML comments <!-- --> as they are ignored by pandoc –  Bordaigorl Sep 3 '14 at 21:40

# Proof of concept

The following code implements a "proof of concept" showing that the approach (1) I proposed in another answer is feasible.

This example defines a Markdown environment which dumps its contents verbatim to an auxiliar file (called \jobname-aux.md), and inmediatelly uses Lua to parse that file as markdown. For this parsing it uses the library markdown.lua, which currently can only output HTML.

The result of the parsing is split into lines, and each line fed again to TeX via tex.print function. Currently those contents are inserted in a verbatim environment, because they are HTML.

# MWE

\documentclass{article}
\usepackage[margin=2cm]{geometry}
\usepackage{fancyvrb}
\usepackage{fontspec}
\setmonofont{Consolas}\fontspec{Consolas}
\usepackage{markdown}

\title{Proof of concept}
\author{JL Diaz}

\begin{document}
\maketitle
\begin{Markdown}
# Idea

This is a _proof of concept_ about the posibility of using
a [markdown][1] parser
embedded in LuaTex.

## How it works
The contents of a markdown environment are written verbatim
in an auxiliar file, which is later processed by Lua (using
[markdown.lua][2] code,
and the result inserted back in TeX.

# Status
Currently markdown.lua only can generate HTML output, so
the result is inserted back in a Verbatim environment into
the TeX document. When proper LaTeX output would be available
the Verbatim will be removed.

[1]: http://daringfireball.net/projects/markdown/
[2]: http://luaforge.net/projects/markdown
\end{Markdown}
\end{document}


After compiling it with lualatex the resulting pdf shows the HTML translation made by Lua in a verbatim environment:

# Setup

In order to reproduce this result, you will need the following files:

## markdownify.lua

These are its contents:

function splitIntoLines(str)
local t = {}
local function helper(line) table.insert(t, line) return "" end
helper((str:gsub("(.-)\r?\n", helper)))
return t
end

function markdownify(f)
file=io.open(f)
file:close()
s = markdown(s)
t = splitIntoLines(s)
tex.print("\\par")
tex.print("\\begin{verbatim}")
tex.print(t)
tex.print("\\end{verbatim}")
end


## markdown.sty

These are its contents:

\directlua{dofile("markdown.lua")}
\directlua{dofile("markdownify.lua")}
\makeatletter
\def\Markdown{\FV@Environment{}{Markdown}}
\def\FVB@Markdown{%
\@bsphack
\begingroup
\FV@UseKeyValues
\FV@DefineWhiteSpace
\def\FV@Space{\space}%
\def\FV@ProcessLine{\immediate\write\FV@OutFile}%
\immediate\openout\FV@OutFile \jobname-aux.md\relax
\let\FV@FontScanPrep\relax
\let\@noligs\relax
\FV@Scan}
\def\FVE@Markdown{\immediate\closeout\FV@OutFile\endgroup\@esphack\markdownify}
\DefineVerbatimEnvironment{Markdown}{Markdown}{}
\makeatother
\def\markdownify{\directlua{markdownify("\jobname-aux.md")}}


# Update

I coded a new version of the .lua and .sty files which does not create any auxiliar file, but instead process all "in memory". Thanks to mbork which pointed me to this answer which shows a technique to record the contents of an environment, and have those contents in a string when the environment ends.

I adapted that answer to this problem. The tricky part was avoiding that the environment contents were seen by tex. In the referred answer, in addition to record the contents they are still part of the document, but we want to avoid this in this case. However, if I delete those contents while I'm recording them (so that TeX cannot see them), then TeX won't be see the \end{Markdown} either and all will fail. So I had to hardcode that particular line.

These are the new files which implement this solution:

## markdownify.lua

local mybuf = ""
if buf:find("\\end{Markdown}") == nil then
mybuf = mybuf .. buf .. "\n"
return ""
else
return nil
end
end
function startrecording()
end

function stoprecording()
markdownify(mybuf)
end

function splitIntoLines(str)
local t = {}
local function helper(line) table.insert(t, line) return "" end
helper((str:gsub("(.-)\r?\n", helper)))
return t
end

function markdownify(s)
s = markdown(s)
t = splitIntoLines(s)
tex.print("\\par")
tex.print("\\begin{verbatim}")
tex.print(t)
tex.print("\\end{verbatim}")
end


## markdown.sty

(Much simpler now!)

\directlua{dofile("markdown.lua")}
\directlua{dofile("markdownify.lua")}
\newenvironment{Markdown}{\directlua{startrecording()}}{\directlua{stoprecording()}}


The MWE can compile without changes, but fancyvrb is not longer required (I used it to dump the contents of the environment to an auxiliar file).

-

You may wanna try kramdown, which has the ability to convert your markdown files to LaTeX source file:

kramdown -o latex --template document some.markdown >some.tex

-

Maybe not exactly what you are looking for. Take a look at Uwe Lück's nicetext package. There's a wiki.sty, which "en­ables Wiki-like markup to re­place some LaTeX markup."

I never used it, so judge yourself! ;-)

-

Think of this not as an answer to the question (since the easy answer is no, no such implementation exists, and yes,LaTeX is Turing complete) but as a bit of research for anyone thinking of implementing a solution.

If you consider Markdown's origin, then it seems natural to turn to Perl for a solution. Sadly, while there are Markdown parsers in Perl, all of their output is HTML. That said any reasonably competent Perlesta should be able to make the necessary fork and adapt/extend the module of their choice.

It could be any of these as a target to extend, I've no further knowledge as I've yet to look at the code in question. And while it may be obvious it bears stating that the 'extender' needs to be fluent in both LaTeX, Perl and Markdown.

-
Reading this suggests that I'm such a person. !!##%\$^& idiot---I really need more shiny objects ;) –  hsmyers Mar 10 '13 at 18:21
Being no expert in Markdown (nor Perl), I'm wondering whether it's Perl which is natural (which would be strange, given Markdown's nice syntax:P), or regexen? –  mbork Mar 10 '13 at 18:27
@mbork I'm not sure you can be an expert in Markdown, but I will admit to sufficient expertise in Perl to tackle the problem. Which same is made an order of magnitude easier by starting with an existing converter. Perl is one of the best languages to convert text from one form to another. It also has a long history with LaTeX. For instance if the source was POD (Perl Documentation language) then the task would already nearly be done. The hard part is embedding the necessary in PerlTeX, but that is where the fun would be! I would use whatever the fork uses, could be real parser or regex :) –  hsmyers Mar 10 '13 at 18:33