# Different approach to literate programming for LaTeX

I have spent a couple of hours rethinking the literate style tools we have for LaTeX, which are essentially doc, docstrip and ltxdoc. Although the traditional workflow with a good editor can be less of a pain, imprisoning the code in guards and percentage marks can be disruptive to the flow of writing code, especially to newcomers. With this in mind and also to try and improve the typography and the functionality of the code, I came up with an alternative.

The central idea is to be able to use any class as the base class (in the MWE I have used the tufte-book) and to be able to have pretty printing using a package such as listings. The approach is to use only one .tex file which when compiled produces the documentation as well as the class, package or files.

Essentially it saves code verbatim when is defined. The MWE example only outputs to one .sty file, but it is possible to save to as many files as one needs (simulating the docstrip guards). A small installation script can be incorporated and self generated with LuaLaTeX if one wants to also simulate docstrips ability to install files in different directories.

What precautions should one take in developing such a code and are there any essential features of the doc/docstrip approach that cannot be incorporated in my approach?

\documentclass{tufte-book}
\usepackage{microtype,soul}
\makeatletter
\usepackage[charter]{mathdesign}
\def\rmdefault{bch} % not scaled
\def\ttdefault{blg}
\usepackage{xcolor,filecontents,ragged2e}
\usepackage[listings, theorems]{tcolorbox}
\definecolor{theblue} {rgb}{0.02,0.04,0.48}
\definecolor{thered}  {rgb}{0.65,0.04,0.07}
\definecolor{thegreen}{rgb}{0.06,0.44,0.08}
\definecolor{thegrey} {gray}{0.5}
\definecolor{theframe}{gray}{0.75}

% Emphasis
emphstyle={\ttfamily\textcolor{#1}}}}%

\lstset{language={[LaTeX]TeX},
escapeinside={{(*@}{@*)}},
numbers=left, gobble=2,
stepnumber=1,numbersep=5pt,
numberstyle={\footnotesize\color{gray}},firstnumber=last,
breaklines=true,
framesep=5pt,
basicstyle=\small\ttfamily,
showstringspaces=false,
% keywordstyle=\ttfamily\textcolor{thegreen},
stringstyle=\color{orange},
breakatwhitespace=true,
showspaces=false,  % shows spacing symbol
xleftmargin=0pt,
xrightmargin=5pt,
aboveskip=3pt plus1pt minus1pt, % compact the code looks ugly in type
belowskip=7pt plus1pt minus1pt,  % user responsible to insert any skips
}
\lst@RequireAspects{writefile}

\lstnewenvironment{Macro}[1][Test]
{\emphasis{#1}\marginpar{\vskip0.5\baselineskip\footnotesize\color{thered}\texttt{\string#1}}
\lst@BeginAlsoWriteFile{textsamples.sty}}
{\endgroup}

\newcommand\lorem{Fusce adipiscing justo nec ante. Nullam in enim.
Pellentesque felis orci, sagittis ac, malesuada et, facilisis in,
ligula. Nunc non magna sit amet mi aliquam dictum. In mi. Curabitur
sollicitudin justo sed quam et quadd. \par}

\DeclareRobustCommand{\todo}[1]{\sidenote{\hl{#1}}}
\author{Y Lazarides}\publisher{Camel Press}
\title{\parindent0pt A New Approach in\\ Documenting Macros}
\begin{document}

\backmatter
\maketitle
\parindent0pt

\mainmatter

\tableofcontents

\chapter{The Approach}
\section{User Documentation}
The user documentation can be styled a bit better. The user should be able to
write the package or class in a more natural style.\todo{Write a few more notes here.}
\medskip

\begin{tcblisting}{colframe=thegreen,boxrule=1pt,colback=thered!5,listing options={style=tcblatex}}
The macro  \verb+\LaTeXe+, typesets the \LaTeXe logo.
\end{tcblisting}

\section{Implementation}

\lorem

\Macro[\ProvidesPackage]
\NeedsTeXFormat{LaTeX2e}
\ProvidesPackage{textsamples}[2012/02/13 v1.0 sample texts]
\endMacro

\lorem

\begin{Macro}[\lorem]
\newcommand\lorem{\leavevmode
Fusce adipiscing justo nec ante. Nullam in enim.
Pellentesque felis orci, sagittis ac, malesuada et, facilisis in,
ligula. Nunc non magna sit amet mi aliquam dictum.
In mi. Curabitur sollicitudin justo sed quam et quadd.
\par
}
\end{Macro}

\lorem

\emphasis{\tex}
\begin{Macro}[\tex]
\gdef{\tex}{%
\TeX\xspace (*@\sidenote{You can even place sidenotes.%
These notes have no impact on the %
code being written to the file.} @*)
}
\end{Macro}
% closes the open file
\begingroup\lst@EndWriteFile

\section{Full listing of {textsamples.sty}}
\lstinputlisting{textsamples.sty}
\end{document}


Note, I have used listings for the verbatim writes and tcolorbox for the example self-running box. It also omits doc's many useful macros which would still need to be adapted.

-
Note that you have to save your sample code as textsamples.tex to make it work. –  cjorssen Mar 8 '12 at 12:19
Very nice approach. I always thought that a listings (pretty printing) support was missing to the doc/docstrip mechanism. –  cjorssen Mar 8 '12 at 12:19
@cjorssen Thanks for picking up the \jobname. I've removed a lot of code to make a MWE. Hardcoded it in the example. –  Yiannis Lazarides Mar 8 '12 at 12:24
Gorgeous! You may want to reset the line numbering before writing the full listing. –  ienissei Mar 8 '12 at 15:13
@ienissei Oops! Also at the full listing one should switch style, as I emphasize the macros as they were defined. Thanks for picking it up. –  Yiannis Lazarides Mar 8 '12 at 15:42

As discussed in earlier in the comments, it is quite possible to solve the problem of defining classes with this method. Granted, as you said, it takes two runs, but it is possible to make a document that will deal with the writing (and some minimal printing) on the first run, and that will process class-defined commands on the second run only (if correctly marked up, obviously).

In order to perform this check, I rely on the existence (or lack thereof) of an .aux file, because it is both easy – most IDEs can delete it in one click, it allows for overwrites, etc. – and user friendly in case of problem – as I expect most users to try to delete this file first if something goes wrong.

The only requirement is that we store the code in a class file instead of a package. The following MWE implements this solution, and also a macro-based solution to the problem of multiple files (you should write one file after the other, not mix them up haphazardly). Due to the fact that I am using comma-separated lists for the macros, it might make things easier to index, e.g. adding an optional argument for the type of thing created (macro, environment, counter, etc.).

The .cls file:

\NeedsTeXFormat{LaTeX2e}
\ProvidesClass{literate}

\newif\if@firstrun
\@firstruntrue
\newif\if@iscsname
\@iscsnamefalse

\IfFileExists{\jobname.aux}
{\IfFileExists{\jobname.cls}
\IfFileExists{\jobname.sty}
{\RequirePackage{\jobname}}{}
\@firstrunfalse}

\newcommand{\filetype}[1]{%
\if@firstrun
\begingroup\lst@EndWriteFile
\fi
\def\usefiletype{#1}
}

\section{Listings for \jobname.\usefiletype}%
The file contains the following code:
}

\RequirePackage{etoolbox,xstring,xcolor,listings,verbatim,docmute,makeidx}
\lst@RequireAspects{writefile}
\makeindex

\newcommand{\numbercolor}{\color[HTML]{655643}}% Ash
\newcommand{\macrocolor}{\color[HTML]{78948D}}% Teal
\newcommand{\commentcolor}{\color[HTML]{BF4D28}}% Saffron
\newcommand{\mathcolor}{\textcolor[HTML]{CF872E}}% Ochre
\newcommand{\emphcolor}{\color{cyan}}
\newcommand{\identifiercolor}{\color{black}}
\newcommand{\bracketcolor}{\color{black}}

\lstset{
language=[AlLaTeX]TeX,
breaklines=true,
columns=spaceflexible,
emptylines=0,
breakindent=1em,
tabsize=4,
basicstyle=\ttfamily,
texcsstyle=*\macrocolor,
identifierstyle=\identifiercolor,
\end{code}

It can be used this way:

\begin{macro}
\mymacro[format]{text}
\end{macro}

And outputs this:

\begin{example}
\mymacro[\emph]{some text}
\end{example}

Then we create a counter:

\begin{code}[c]{something}
% Some comment
\newcounter{something}
\end{code}

\section{The second section}

\filetype{sty}

We also create an |.sty| file:

\begin{code}{}
\NeedsTeXFormat{LaTeX2e}
\ProvidesPackage{MWE}
[2011/03/20 v1 Some sample package]
\end{code}

\pagebreak A new environment:

\begin{code}[e]{myenv}
\newenvironment{myenv}{}{}
\end{code}

A new color:

\begin{code}[col]{ochre}
\usepackage{xcolor}
\definecolor{ochre}{HTML}{CF872E}
\end{code}

\begin{example}
\textcolor{ochre}{Some ochre text!}
\end{example}

\printlistings

\printindex

\end{document}


Requires 2 runs of LaTeX to print the documentation + one run of MakeIndex and one final run of LaTeX.

Edit (April 3): Added an AfterEndEnvironment hook so the new macro names are set globally after the first code environment (fixes a bug in syntax highlighting).

Edit (April 10): Added several new things:

• A macro for adding some control sequences to listings' colouring, without indexing them: \addcs{}, which takes a CSV list as its argument.

• An environment that writes a copyright file and prints it at the beginning of every new file: \begin{copywrite}.

• A command that inputs the listings of all created files, with a "heading" that can be customised via \renewcommand{\listingshead} – by default, I have set it to print a section title and a short sentence. (Could be made more customisable I suppose…)

• Semi-automatic indexing (see below).

• Some error messages, because wrong indexing commands are easy to produce and the LaTeX error message was really, really unhelpful.

Several macros that automatically index the marked content of every code environment based on its type. It also works with @ characters in macro names. By default, it is assumed that the string is a macro but you can change it using the optional argument (e.g. \begin{code}[e]{myenv} for an environment).

I have defined macros (m, default), environments (e), floats (f), lengths (l), dimensions (d), counters (c), packages (p) and options (o). New types can be defined implicitly:

\begin{code}[great-macro]{mygreatmacro}


in which case they will be treated as macros. They can also be defined explicitly in the preamble with:

\addtoindex{<short name>}{<name to index>}{<format>}{<indexing code>}
% The starred version is for control sequences, the normal version for everything else
% Helper macros are \macroname, \entryname and \entrytype


In the main file, you then need to write \docindex{mymacro} for control sequences, or \docindex[c]{mycounter} for everything else. This is made necessary by the fact that, while you can have only one control sequence with a given name, you can have several strings of other types with the same name (say the \chapter command vs the chapter counter and perhaps a chapter colour).

I will try to set up a repository with the complete file (which includes options at class selection), and try to work on some documentation as things are getting a bit complex now.

And I would indeed be very happy to get any feedback regarding bugs and possible features or improvements – in the hope that it can become a package once tested and cleaned up.

-
Thanks for the solution of the class issue. Will study it a bit more carefully tomorrow. –  Yiannis Lazarides Apr 1 '12 at 17:10
@YiannisLazarides Thanks. Let me know if you find any problem with it. I am working on a solution regarding indexes (preferably defining the type of macro we are indexing once, and then having one macro to index everything). I will edit my answer when it is working. –  ienissei Apr 3 '12 at 20:52
So far looks very promising and I added a forthcoming bonus for you:) –  Yiannis Lazarides Apr 3 '12 at 20:59
@YiannisLazarides Wow, very nice :) I will try to get the indexing to work in the next few days. I can parse things correctly at the moment, but I still need to do some string replacement, formatting and to load the indexing engines. –  ienissei Apr 3 '12 at 21:07
Thanks! Indexing is not the easiest things to get right and is about the only thing still missing so far. –  Yiannis Lazarides Apr 3 '12 at 21:11

Once a technology gets entrenched it is difficult to displace and doc/docstrip has been working well and served the community for many years.

The approach as listed above has certain limitations:

Use of the defined class/package. If used to define a class, the class itself cannot be used to typeset the code and the manual. This might be possible on two runs. Another possibility is to first typeset the code and its comments only, following it with the "user manual". Note that there is no "driver" in the approach used. So commands are available in a sequential order as they are entered and saved in the file.

Multiple files. One of the advantages of the doc/docstrip is that code can be saved in multiple files. The example above can be extended to cater for this.

          \lstnewenvironment{Macro}[1][Test]
...
\lst@BeginAlsoWriteFile{textsamples.sty}}


Instead of hardcoding the filename in this case textsamples.sty, this can be passed onto Macro. It is also possible to extend the code to accept a list of files and or possibly to have special predicates similar to doc/docstrip.

Automating indexing. This will be hard to program as the contents of the Macro are not parsed fully. However, indexing commands can easily be added, so that when a macro is added it is indexed. This will also eliminate the problem of overindexing and having to exclude commands from the index. Rather it leaves it to the user to add commands as necessary.

I have used a slightly modified approach to the above to document JavaScript code and so far it has proved very useful.

-

There is other literate programming tools that can be used to prepare TeX macros. I have experience with using noweb (Norman Ramsey's web) but have heard about another option muweb. Both noweb and muweb provide a language agnostic implementation of cweb and ctangle.

Since noweb is language agnostic it provides an extension mechansim, so that external tools can be used to produce the list of macros used in some code snippet. (I think there is some available, but I am not sure I used it).

If you use noweb to produce LaTeX macros it supports:

• Using the defined class to prepare the documentation.
• Multiple file (look up for roots in the documentation).
• Automated indexing (with an appropriate filter).
• Generation of HTML documentation, instead of LaTeX, which is very nice for online documentation.
• Use of code snippets, so that you can avoid \input statments if you want to provide easy to use files (for instance, if you send your files to a journal or on the arXiv).

If you are interested to see how the output look like, I have written a complete TeX format using this, it is called Bhrìd TeX and is free software. Example output is found in the download section, the file is called bhridman.pdf (in French).

I also have useful Makefile macros to ease the use of NOWEB, they are called BSD Make Pallàs Scripts and the file you are looking for is noweb/noweb.latex.mk.

-