Automating extraction of Markdown-formatted tables from an (incomplete) LaTeX document

Question

I am trying to automate the presentation of my note taking. I would like to be able to regex all Pandoc tables in my document, add LaTeX around them, compile them and show the result in a PDF.

Process flow for my desired automatic Selection, Addition LaTeX and Compilation:

1. Select/extract all Pandoc tables with a regular expression (discussion here about developing one).
2. Add head and tail LaTeX code around the tables (discussion about this topic started here).
3. Compile to PDF.

My current process for this workflow is entirely manual:

1. Select some part of text and paste to a new document;
   • iterate for the next Pandoc table until the end.
2. Run pandoc -f markdown -t latex on the selection in suitable editor (f.e. Textmate).
3. Add manually surrounding "begin" (\usepackage{caption}; \usepackage{longtable}; \usepackage{booktabs}) and "end" (\end{document}) LaTeX snippets.
4. Manually press CMD-R to compile the document and see the result.

Example of such file with Markdown markup of Pandoc tables inside LaTeX and as a link here:

\section{Aortic valve stenosis}
everything - etiology, symptoms, diagnostic tests, grades of aortic valve stenosis

Aortic regurgitation. 
The murmur of aortic regurgitation occurs during diastole as the aortic valve fails to lose completely and blood regurgitates from the aorta back into the LV. 
Murmur - high-pitched decrescendo murmur heard best along left lower sternal border. 

------------------------------------------------------
Age < 70 years (n=324)      Age >70 years (n=322)
--------------------------- --------------------------
Bicuspid (50\%)             Bicuspid (27\%)

Postinflammatory (25\%)     Postinflammatory (23\%)

Degenerative (18\%)         Degenerative (48\%)

Unicommissural (3\%)

Hypoplastic (2\%)           Hypoplastic (2\%)

Indeterminate (2\%)
------------------------------------------------------

Table: Common causes of Aortic Stenosis Among Patients requiring Surgery. 
% http://emedicine.medscape.com/article/150638-overview#aw2aab6b2b4aa


----------------------------------------------------------------------------
                            Aortic      Mild        Moderate    Severe
                            sclerosis   
--------------------------- ----------- ----------- ----------- ------------
Aortic jet velocity (m/s)   ≤2.5 m/s    2.6-2.9     3.0-4.0     >4.0

Mean gradient (mmHg)        -           <20 (<30)   20-40b      >40b (>50a)
                                        (30-50a)    

AVA (cm2)                   -           >1.5        1.0-1.5     <1

Indexed AVA (cm2/m2)                    >0.85       0.60-0.85   <0.6

Velocity ratio                          >0.50       0.25-0.50   <0.25
---------------------------------------------------------------------------


\section{Mitral regurgitation} 
% Etiology, symptoms, diagnostic tests and treatment. 
% Grades of mitral regurgitation. 

Mitral regurgitation (MR) is defined as an abnormal reversal of blood flow from the left ventricle (LV) to the left atrium (LA).
It is caused by disruption in any part of the mitral valve (MV) apparatus. 
The most common etiologies of MR include MV prolapse (MVP), rheumatic heart disease, infective endocarditis, annular calcification, cardiomyopathy, and ischemic heart disease.

---------------------------------
grade           None or trace MR
--------------- -----------------
1               trivial

2               mild

3               moderate

4               severe
----------------------------------

Table: Gradient of colorjet profiles in MR


---------------------------------------------------------------------------
                                Mild        Moderate        Severe
------------------------------- ----------- --------------- --------------- 
Valve area                      > 1.5       1.0-1.5         <1.0

Mean gradent mmHg               <5          5-10            >10

Pulmonary artery pressure mmHg  <30         30-50           >50
---------------------------------------------------------------------------

Table: Classification of mitral stenosis severity. At heart rates between 60-80 bpm in sinus rhythm. 

%% Classification of Mitral valve stenosis
% Wilkins score Grades 1-4. 
% http://www.echopedia.org/wiki/Classification_of_valve_stenosis_and_regurgitation



Mitral regurgitation is a dynamic process; the magnitude of regurgitation differs in early, mid, and late systole.

hp_sep02_valve.pdf

% https://www.youtube.com/watch?v=O0EapRNaNJ0
% hp_sep02_valve.pdf

\subsection{ECHO diagnostics}
Check visual valve assessment by ECHO 

\subsection{Color doppler}


-----------------------------------------------------------------------------
                        Mild                 Moderate         Severe
----------------------- -------------------- ---------------- ---------------                       
Vena contracta (mm)     < 3                  3—6.9            ≥ 7

Jet area (\%)           Small central jet    Variable         Large central 
                        (<20\% of LA area)                    jet (\%gt;40\% 
                                                              of LA area)
-----------------------------------------------------------------------------

Table: Quantification based on Color doppler in Mitral Rerurgitation. 


To examine heart or blood vessels. 
ECHO - high f sound waves. 
Doppler technology allows determination of speed and direction of blood flow by utilizing Doppler effect. 

\subsection{Vena constracta}

Figures. 
12.7.1 vena contracta.png
12.7.1 Jet_Components.png

Vena contracta: The vena contracta corresponds to the region in which blood passes through the valve. Velocity is highest here. The width of the vena contracta is a good marker of the severity of mitral regurgitation because it corresponds to the diameter of the regurgitant orifice area. A diameter exceeding 7 mm indicates severe regurgitation. However, like all distance measurements it is limited by two facts: a) regurgitant orifices may have many geometric shapes, b) quite often more than one jet is present. Nevertheless, the vena contracta is an important clue to the severity of regurgitation.

Vena constructa is the point in a fluid stream where the diameter of the stream is the least, and fluid velocity is at its maximum. 
In ECHO, its measurement describes the smallest area of the blood flow jet as it exits a heart valve. 
This corresponds to the Effective Orifice Area calculated for heart valves using the continuity equation.

With echocardiography it is easy to detect mitral regurgitation. 
In contrast, quantification of mitral regurgitation is much more difficult. 
It calls for considerable experience. 
No single method or feature can be used to fully describe the severity of mitral regurgitation. 
All components of quantification must be considered; an integral approach should be adopted. 
In addition, it is important to understand the hemodynamics and sequelae of mitral regurgitation. 
Put all the pieces of the puzzle together and interpret your findings in the context of clinical findings. 
Ultimately, the severity of mitral regurgitation determines whether one opts for a surgical, an interventional (e.g. the MitraClip procedure) or a conservative approach.




------------------------------------------------------------------------------------
Qualitative (visual assessment)     Semiquantitative         Quantitative
----------------------------------- ------------------------ -----------------------
Jet size (vena contracta,           Size of vena contracta   PISA method
flow convergence), ECHO

LV size (and function),             Size of flow             Volumetric method
LV volume overload?                 convergence zone

PA pressure                         Jet area    

(MV morphology)     

(left atrial size)      

(other indirect signs)      
-----------------------------------------------------------------------------------

Table: Quantification of mitral regurgitation. 
% https://123sonography.com/node/20867




\subsection{PISA}

The PISA (proximal isovelocity surface area) method employs the proximal flow convergence zone to measure the volume of regurgitation

The principle underlying this method is straightforward: the flow convergence zone corresponds to regurgitant flow. Blood flow velocity increases as it approaches the regurgitant orifice. The proximal flow convergence zone can thus be described as "hemispheric shells" in which the velocity on the surface of each of the shells is equal. The quantity of blood flow (regurgitant flow) can be calculated when the radius of the shell and velocity at its surface are known:


Regurgitant flow is 
\begin{equation}
Q = 2 \pi r^{2} v_{Nyquist}
\end{equation}
where $V_{Nyquist}$ is aliasing velocity, and $r$ is PISA radius of convergence zone. 

With the PISA method we use the shell where aliasing occurs - where the color abruptly switches from a distinct blue or red to turbulent (multicolored) flow. Velocity (aliasing velocity) can be determined exactly at this site. We can also measure the radius (r) of the hemisphere at this site.

12.7.3 EROA.png

Calculation of EROA (PISA)
% EROA = Effective Regurgitant Orifice Area. 
% Calculated by instantaneous regurgitant flow and derives from: 
\begin{equation}
EROA = 2 \pi r^2 \frac{V_{alias}}{V_{max}}
\end{equation}
where obtaining the maximum velocity across the MR jet using continuous-wave doppler. 
% http://www.wikiecho.org/wiki/Proximal_isovelocity_surface_area 

Based on the principle of conservation of mass, the PISA method also permits us to measure the effective regurgitant orifice (the functional size of the "hole"). To perform this calculation we have to know the peak velocity of the MR signal. This is achieved by obtaining a CW Doppler spectrum through the mitral regurgitant jet.


------------------------------------------------------------
                    Mild    Moderate            Severe
------------------- ------- ------------------- ------------            
PISA Radius         <0.4                        >1.0
Regurgitant 

volume (ml/beat)    <30     30—44. 45—59        ≥ 60

EROA (cm2)          <0.2    0.2—0.29. 0.3—0.39  ≥ 0.4
------------------------------------------------------------

Table: PISA reference values. 
% https://123sonography.com/node/20867

\subsection{Auscultation}
% since we are talking about valves
hp_sep02_valve.pdf

Diaphragm of stetoscope in supine, left decubitus and sitting positions. 

How can I automate the selection of all Markdown Pandoc tables, then addition LaTeX and run the compilation to output PDF?

Answer 1

The following command may surprise you, because it achieves in a single pipeline what you want (or at least, it achieves what I think that you want).

The command fits into a single Tweet even :)     :

pandoc test_dummy.tex -f markdown -t html | grep -E '(^<|^$|^ *$)' \
| grep -v "^<p" | pandoc -f html -o tables.pdf

Or, to format it a bit more nicely:

pandoc                   \
     test_dummy.tex      \
    -f markdown          \
    -t html              \
                         \
| grep -E '(^<|^$|^ *$)' \
                         \
| grep -v '^<p'          \
                         \
| pandoc                 \
    -f html              \
    -o tables.pdf        \
    --latex-engine=xelatex

_{(It turned out that I had to use xelatax for the PDF generation -- your source file contained a few Unicode characters which triggered an error when processed via the default engine, pdflatex.)}

Explanation:

1. Your input file isn't really a valid LaTeX document (despite of the .tex suffix in the file name). It is a mixture of LaTeX and Markdown snippets.

2. First pandoc command in pipeline:
   This may surprise you: I declared -f markdown to tell Pandoc that it should treat the input file as Markdown (not as LaTeX!)..

   The reason for this trick is this: Pandoc tolerates raw_tex snippets sprinkled into the Markdown sources. This is good for when you want to create LaTeX-based output documents (Beamer, PDF, LaTeX itself...). Pandoc would then let pass these raw TeX snippets though to LaTeX unchanged. -- For non-LaTeX output, these snippets are simply ignored... since you do not care about these (for your current purpose) anyway, we can ignore anything that goes wrong here.

   The resulting output will have HTML format due to the -t html parameter. Almost all content will be tagged in HTML tags, apart from some lines which consist of text strings alone. All lines (other than the mentioned "text only strings") will start with <.

3. First grep command in pipeline:
   This keeps only these lines which...

   • ...either start with a tag: ^<,
   • ...or are completely empty: ^$,
   • ...or contain only blanks: ^ *$.

   So text snippets without any tag will be

4. Second grep command in pipeline:
   This throws out all lines which contain text wrapped in paragraph tags and hence start with <p.

   What's now left should be HTML code only which is devoted to table rendering. (That HTML code is without the <html><head> ... </head><body> ... </body></html> overhead).

5. Second pandoc command in pipeline:
   The stage is now prepared for another Pandoc command: this takes the HTML code representing the left-over (ex-Markdown) tables from the previous command, and converts that into a PDF document.

Voilà!

Here is a screenshot of the result:

---

Caveats: While my command works for the sample file you provided, it may require further tweaking if you run it against other similar files, which may contain additional elements which I may not have considered...

---

Update

In order to remove the commented out lines starting with a % sign (as requested by OP in a comment), use this modified command:

pandoc                              \
     test_dummy.tex                 \
    -f markdown                     \
    -t html                         \
                                    \
| grep -E '(^<|^$|^ *$)'            \
                                    \
| grep -v '^<p'                     \
                                    \
| sed 's#%.*</caption>#</caption>#' \
                                    \
| pandoc                            \
    -f html                         \
    -o tables.pdf                   \
    --latex-engine=xelatex