Converting Markdown to LaTeX, in LaTeX

Question

I've started to use markdown for my notes simply because its faster than writing latex, but I would prefer to have the extra power to LaTeX. Is there/would it be possible to have a markdown environment which converted the markdown to latex? Is LaTeX powerful enough to parse another language?

Side question: Is LaTeX Turing complete?

I guess one option would be to pass the data to a script I wrote to convert them. I have read this is possible, is it possible to have latex pipe data into the stdin for another program?

Answer 1

A proper markdown parser is a task too complex for latex. Not because TeX is not a Turing complete language (it is), but because it would be very difficult to implement, and probably will have a very poor performance.

One idea which immediately comes to mind is to use LuaTeX, and code the markdown parser in Lua language. This sound certainly feasible.

According with Wikipedia, there are only three Markdown parsers implemented in Lua:

• markdown.lua. It is simple, and it would be probably easy to integrate into a LuaLatex package, but currently it only outputs HTML, so a complete rewrite of the output part should be done to generate latex code instead.

• lunamark. This is very complete. It can output a variety of formats, including LaTeX, and supports a lot of non-standard extra modules. However, it has a number of dependencies of other Lua packages, so in order to integrate it with LuaTex it should be stripped down (for example, template support could be removed, since the template will be the document which uses it, and also all output formats except LaTeX).

• lua-discount This is a lua binding to a parser written in C, so it can be discarded because that would not be embeddable in LuaTeX.

So the Lua approach basically reduces to one of those two:

1. Change markdown.lua to make it output LaTeX
2. Strip down lunamark to remove all unnecessary stuff.

I'm not sure which one will be easier. Probably 1 will produce some useable in a shorter time, but 2 looks like a better long term solution.

Also, the interface between LaTeX and Lua should be defined. A sensible approach would be:

% Latex stuff....
\begin{markdown}
# First section
etc..
\end{markdown}

However I'm very new to the LuaTeX world, and (currently) I don't know how to pass to a Lua function all the text inside the markdown environment. Also we should prevent TeX to tokenise all that text.

A possible idea is to use some verbatim-like tricks to write the contents of the environment to a file, and then use lua to process that file and insert back the resulting tex in the main document (a tex.sprint() would do, wouldn't?)

I would like to hear the opinion of the LuaTeX experts here...

PS: Does this count as an answer? Or should I delete it and re-post as a question instead?

Answer 2

I'm not sure why a Pandoc-based solution/answer on this page harvested so many upvotes, while being overly complicated.

If you use Pandoc, the method is far easier:

1. Just write Markdown.
2. At occasions where you want to apply LaTeX features in your final document, just sprinkle the Markdown with your LaTeX snippets...

Here is a working (not so minimal) example, where the main contents are written in Markdown, with the occasional LaTeX code sprinkled in between:

% Proof of Existing Pandoc Feature
% Kurt Pfeifle
% May 25th, 2015

# Basics

This is not just a *proof of concept*, but a basic utilization of [Pandoc][2]'s behavior
when it comes to [Markdown][1] processing.

* Pandoc, by default, passes any \LaTeX\ code snippets it identifies within the Markdown
  source file to the target document, if that target document is a \LaTeX\ one (this
  includes f.e. Beamer or PDF output.
  (It does not pass these snippets to any other output formats, but instead drops them.)
* Pandoc, by default, also passes any HTML code snippets it identifies within the
  Markdown sources to the target document, should that be HTML based (this includes f.e.
  EPUB or RevealJS output).
  (It does not pass these snippets to any other output formats, but instead drops them.)
* This allows for any specific formatting to be achieved in the target document format:
  ***(1)*** Insert two versions of the snippet, one as HTML, one as \LaTeX\. ***(2)***
  The HTML one will make it to the HTML-based targets, while \LaTeX\ is being dropped;
  the \LaTeX\ will make it into \LaTeX-based targets, while the HTML is being dropped.

It also supports linked references. **[Click here](#tab:fsttable)** to jump to a page with a
table.


## How it works

It works out of the box:

1. Just write a Markdown document, and sprinkle your \LaTeX\ code snippets in between.
1. Save the document with an `.md` suffix.
1. Run the Pandoc conversion:

    ```` {.bash}
     pandoc --from=markdown --output=my.tex my.md --to=latex --standalone
    ````

    or

    ```` {.bash}
     pandoc --from=markdown --output=my.pdf my.md                                   \
            --variable=geometry:"margin=0.5cm, paperheight=421pt, paperwidth=595pt" \
            --highlight-style=espresso
    ````

I want a few words appear as \textcolor{red}{red} or in a \textcolor{green}{different}
\textcolor{blue}{color}. Here is the Markdown source code of previous sentence; it uses
no syntax highlighting (unlike the previous two code blocks):

     [...] as \textcolor{red}{red} or in a \textcolor{green}{different}
     \textcolor{blue}{color}.

After this paragraph, I want to insert a page break. I'll add `\newpage{}` beneath, on
a line of its own.

\newpage{}

This was the Markdown source code (with context) for the page break preceeding this:

```` {.markdown}
After this paragraph, I want to insert a page break. I'll add `\newpage{}` beneath, on
a line of its own.

\newpage{}
````

## Inserting a \LaTeX\ table

Here comes a table. Its code is inserted as \LaTeX\ code into the Markdown source document:

\begin{table}[h]
\centering
\begin{tabular}{|r|l|}
  \hline
  7C0 & hexadecimal \\
  3700 & octal \\ \cline{2-2}
  11111000000 & binary \\
  \hline \hline
  1984 & decimal \\
  \hline
\end{tabular}
\caption\small\textit{\textcolor{magenta}{This table shows some data}}
\label{tab:fsttable}
\end{table}

This is the Markdown code for the previous table, including its textual context:

```` {.latex}
Here comes a table. Its code is inserted as \LaTeX\ code into the Markdown source document:

\begin{table}[h]
\centering
\begin{tabular}{|r|l|}
  \hline
  7C0 & hexadecimal \\
  3700 & octal \\ \cline{2-2}
  11111000000 & binary \\
  \hline \hline
  1984 & decimal \\
  \hline
\end{tabular}
\caption\small\textit{\textcolor{magenta}{This table shows some data}}
\label{tab:fsttable}
\end{table}

This is the Markdown code for the previous table, including its textual context:
````

## Inserting a \LaTeX\ Formula

To include a mathematical formula in Markdown, enclose it with **`$`** characters like this:

```` {.latex}
 $\frac{n!}{k!(n-k)!} = \binom{n}{k}$
````

The result:

$\frac{n!}{k!(n-k)!} = \binom{n}{k}$

# Status

There are no known (to me) bugs for this feature.

[1]: http://daringfireball.net/projects/markdown/
[2]: http://pandoc.org/

Convert it to PDF with this command:

pandoc --from=markdown --output=my.pdf my.md                                   \
       --variable=geometry:"margin=0.5cm, paperheight=421pt, paperwidth=595pt" \
       --highlight-style=espresso

The result:

Answer 3

Here's Pandoc-based solution. You will have to enable --shell-escape for this to work, since it uses \write18. Depending on what you want, you may need to customize the Pandoc options.

\documentclass{article}

\usepackage{fancyvrb}

\newenvironment{markdown}%
    {\VerbatimEnvironment\begin{VerbatimOut}{tmp.markdown}}%
    {\end{VerbatimOut}%
        \immediate\write18{pandoc tmp.markdown -t latex -o tmp.tex}%
        \input{tmp.tex}}

\begin{document}

\begin{markdown}
# Section

Some text that goes on for a while.

A list:

* Item
* Another item 

\end{markdown}

\end{document}

Answer 4

Proof of concept

The following code implements a "proof of concept" showing that the approach (1) I proposed in another answer is feasible.

This example defines a Markdown environment which dumps its contents verbatim to an auxiliar file (called \jobname-aux.md), and inmediatelly uses Lua to parse that file as markdown. For this parsing it uses the library markdown.lua, which currently can only output HTML.

The result of the parsing is split into lines, and each line fed again to TeX via tex.print function. Currently those contents are inserted in a verbatim environment, because they are HTML.

 MWE

\documentclass{article}
\usepackage[margin=2cm]{geometry}
\usepackage{fancyvrb}
\usepackage{fontspec}
\setmonofont{Consolas}\fontspec{Consolas}
\usepackage{markdown}

\title{Proof of concept}
\author{JL Diaz}

\begin{document}
\maketitle
\begin{Markdown}
# Idea

This is a _proof of concept_ about the posibility of using
a [markdown][1] parser
embedded in LuaTex.

## How it works
The contents of a `markdown` environment are written verbatim
in an auxiliar file, which is later processed by Lua (using
[`markdown.lua`][2] code,
and the result inserted back in TeX.

# Status
Currently `markdown.lua` only can generate HTML output, so
the result is inserted back in a `Verbatim` environment into
the TeX document. When proper LaTeX output would be available
the `Verbatim` will be removed.

[1]: http://daringfireball.net/projects/markdown/
[2]: http://luaforge.net/projects/markdown
\end{Markdown}
\end{document}

After compiling it with lualatex the resulting pdf shows the HTML translation made by Lua in a verbatim environment:

Setup

In order to reproduce this result, you will need the following files:

compat_env.lua

Download it from its source

markdown.lua

Download it from its source, making some changes for Lua 5.2 compatibility:

• Add this line near the top of the file local CE = require 'compat_env'
• modify all occurances of setfenv to CE.setfenv
• change all occurances of unpack to table.unpack

markdownify.lua

These are its contents:

function splitIntoLines(str)
  local t = {}
  local function helper(line) table.insert(t, line) return "" end
  helper((str:gsub("(.-)\r?\n", helper)))
  return t
end

function markdownify(f)
 file=io.open(f)
 s = file:read("*all")
 file:close()
 s = markdown(s)
 t = splitIntoLines(s)
 tex.print("\\par")
 tex.print("\\begin{verbatim}")
 tex.print(t)
 tex.print("\\end{verbatim}")
end

markdown.sty

These are its contents:

\RequirePackage{luatexbase}
\directlua{dofile("markdown.lua")}
\directlua{dofile("markdownify.lua")}
\makeatletter
\def\Markdown{\FV@Environment{}{Markdown}}
\def\FVB@Markdown{%
  \@bsphack
  \begingroup
    \FV@UseKeyValues
    \FV@DefineWhiteSpace
    \def\FV@Space{\space}%
    \FV@DefineTabOut
    \def\FV@ProcessLine{\immediate\write\FV@OutFile}%
    \immediate\openout\FV@OutFile \jobname-aux.md\relax
    \let\FV@FontScanPrep\relax
    \let\@noligs\relax
    \FV@Scan}
\def\FVE@Markdown{\immediate\closeout\FV@OutFile\endgroup\@esphack\markdownify}
\DefineVerbatimEnvironment{Markdown}{Markdown}{}
\makeatother
\def\markdownify{\directlua{markdownify("\jobname-aux.md")}}

---

Update

I coded a new version of the .lua and .sty files which does not create any auxiliar file, but instead process all "in memory". Thanks to mbork which pointed me to this answer which shows a technique to record the contents of an environment, and have those contents in a string when the environment ends.

I adapted that answer to this problem. The tricky part was avoiding that the environment contents were seen by tex. In the referred answer, in addition to record the contents they are still part of the document, but we want to avoid this in this case. However, if I delete those contents while I'm recording them (so that TeX cannot see them), then TeX won't be see the \end{Markdown} either and all will fail. So I had to hardcode that particular line.

These are the new files which implement this solution:

markdownify.lua

local mybuf = ""
function readbuf( buf )
  if buf:find("\\end{Markdown}") == nil then
     mybuf = mybuf .. buf .. "\n"
     return ""
  else
     return nil
 end
end
function startrecording()
  luatexbase.add_to_callback('process_input_buffer', readbuf, 'readbuf')
end

function stoprecording()
  luatexbase.remove_from_callback('process_input_buffer', 'readbuf')
  markdownify(mybuf)
end

function splitIntoLines(str)
  local t = {}
  local function helper(line) table.insert(t, line) return "" end
  helper((str:gsub("(.-)\r?\n", helper)))
  return t
end

function markdownify(s)
 s = markdown(s)
 t = splitIntoLines(s)
 tex.print("\\par")
 tex.print("\\begin{verbatim}")
 tex.print(t)
 tex.print("\\end{verbatim}")
end

markdown.sty

(Much simpler now!)

\directlua{dofile("markdown.lua")}
\directlua{dofile("markdownify.lua")}
\newenvironment{Markdown}{\directlua{startrecording()}}{\directlua{stoprecording()}}

The MWE can compile without changes, but fancyvrb is not longer required (I used it to dump the contents of the environment to an auxiliar file).

Answer 5

You may wanna try kramdown, which has the ability to convert your markdown files to LaTeX source file:

kramdown -o latex --template document some.markdown >some.tex

Answer 6

This question is old, but for future reference there is now the package markdown.

\documentclass{article}
\usepackage[fencedCode]{markdown}

\begin{document}
\begin{markdown}
# Header
## My First Steps
Let's have a tiny list:

-   test
-   test2
    -  `\verbatim`

```
foo.bar(do=True)
```
\end{markdown}
\end{document}

There are many options to modify the complexity of the markdown parsing, e.g., \usepackage[hashEnumerators, definitionLists, footnotes, inlineFootnotes, smartEllipses, fencedCode, contentBlocks, pipeTables, tableCaptions]{markdown} in addition it provides \markdownInput{yourfile.md} to include an external .md file into your document.

Answer 7

Maybe not exactly what you are looking for. Take a look at Uwe Lück's nicetext package. There's a wiki.sty, which "en­ables Wiki-like markup to re­place some LaTeX markup."

I never used it, so judge yourself! ;-)

Answer 8

I'm going to stick my neck out here and suggest a good look at Fletcher Penney's Multimarkdown (MMD), which comes with LaTeX tools. I recently converted some MMD files to Latex pretty smoothly (I guessed successfully with multimarkdown -t latex my_text.txt | cat > my_text.tex although there's a built-in shorter command). I'm aware this doesn't directly answer the question, but MMD is a good toolkit to have if you write in both LaTeX and Multimarkdown, and Fletcher Penney has some neat shortcuts in MMD that output really tidily to LaTeX.

Answer 9

Think of this not as an answer to the question (since the easy answer is no, no such implementation exists, and yes,LaTeX is Turing complete) but as a bit of research for anyone thinking of implementing a solution.

If you consider Markdown's origin, then it seems natural to turn to Perl for a solution. Sadly, while there are Markdown parsers in Perl, all of their output is HTML. That said any reasonably competent Perlesta should be able to make the necessary fork and adapt/extend the module of their choice.

• The Original Markdown
• MultiMarkdown
• Text::Markdown

It could be any of these as a target to extend, I've no further knowledge as I've yet to look at the code in question. And while it may be obvious it bears stating that the 'extender' needs to be fluent in both LaTeX, Perl and Markdown.

Answer 10

Here is an improvement to the solution proposed by G. Poore. It uses counters to generate a unique file name for each temp file. The benefit is the ability to perform reverse search inside markdown content.

\documentclass{article}

\usepackage{fancyvrb}

\newcounter{MarkdownId}

\newenvironment{markdown}%
    {\stepcounter{MarkdownId}%
        \VerbatimEnvironment\begin{VerbatimOut}{tmp\theMarkdownId.markdown}%
    }%
    {\end{VerbatimOut}%
        \immediate\write18{pandoc .latex\string\\tmp\theMarkdownId.markdown -t latex -o .latex\string\\tmp\theMarkdownId.tex}%
        \input{.latex\string\\tmp\theMarkdownId.tex}%
    }

\begin{document}

\begin{markdown}
# Section

Some text that goes on for a while.

A list:

* Item
* Another item 

\end{markdown}

\begin{markdown}
- item 1
- item 2
\end{markdown}

\end{document}