Testing Framework API for LaTeX

As part of an online exam application, we need to automate the evaluation of LaTeX scripts written by the students. Essentially, the script should be evaluated against a set of test cases and if it passes all of them should return passed (true) or should throw the error message back so that the student can retry the problem.

We came across qstest, but I wish to know are there any better solutions out there which can be employed in the web app easily and efficiently. The web app has been developed using Django.

-
 Are you making your students program in LaTeX? Or do they produce a LaTeX file using some other programming language? In both cases, I doubt that you are aiming for the rigt direction: (La)TeX is a very bad programming language (great document markup, though!), and if you are starting from another language, why not test that language? Or is that meant to be a (La)TeX tutorial? – Bruno Le Floch Jan 31 '12 at 21:49 It is a Latex tutorial and documents are produced using the same. – primpop Feb 15 '12 at 7:07

Testing in many languages is done using assertions and test cases as you mentioned in your question. An assertion in LaTeX can be of the general form:

      \def\test#1{\def\res{#1}\ifx\foo\res\else\ERROR\fi}


or just return "Passed" or "Failed". I lean for the latter for long test cases, and if you colour code the results is easy to pick-up the errors.

Code can be simple for true false based on meaning

\newcommand\assert[2][]{%
\ttfamily
\def\result{#2}
\ifx\foo\result\textcolor{green}{Passed}%
\else \textcolor{red}{Failed}\fi
\space Test:\,\stepcounter{tst}\thetst
\detokenize{#1}
\par}


or checking if things have been defined (sometimes tricky).

% Checks if defined can be unreliable
\newcommand\assertdef[2][]{
\ttfamily
\ifdefined#2\textcolor{green}{Passed}%
\else \textcolor{red}{Failed}\fi
\space Test:\,\stepcounter{tst}\thetst
\detokenize{#1}
\par
}


The MWE example below tests some fixed point arithmetic and some Lisp relics from the LaTeX kernel and produced the output for the image above:

\documentclass{article}
\usepackage{xcolor,fp}
\parindent0pt
\makeatletter
\newcounter{tst}
\setcounter{tst}{9}
% asserts true on meaning
\newcommand\assert[2][]{%
\ttfamily
\def\result{#2}
\ifx\test\result\textcolor{green}{Passed}%
\else \textcolor{red}{Failed}\fi
\space Test:\,\stepcounter{tst}\thetst
\detokenize{#1}
\par}
% Checks if defined can be unreliable
\newcommand\assertdef[2][]{
\ttfamily
\ifdefined#2\textcolor{green}{Passed}%
\else \textcolor{red}{Failed}\fi
\space Test:\,\stepcounter{tst}\thetst
\detokenize{#1}
\par
}
\begin{document}
\section{\jobname\\ \today}
\edef\test{\@car 123\@nil}\assert[\@car 123\@nil==1]{1}
\edef\test{\@car{1}23\@nil} \assert[\@car {1}23\@nil==1]{1}
\edef\test{\@car {123}{456}{7}\@nil} \assert{123}
\edef\test{\@carcube1234567\@nil}\assert{123}
\edef\test{\@cdr 123\@nil} \assert{23}
\edef\test{\@cdr {134}{x}\@nil}\assert{x}
\edef\test{\@cdr {134}{{x}}\@nil}\assert{{x}}
\let\test\@nnil\assert{\@nil}
\expandafter\def\expandafter\test\expandafter{\the\toks@} \assert{abc\test x\bar}
\def\xx{456}
\def\test{123}\@cons\test{\xx78}\assert{123\@elt45678}
\@cons\test{\xx780}\assert{123\@elt45678}
% assert if defined
\assertdef\assert
\assertdef[\@@par is defined]\@@par
% asserts for fp values

One can get more creative and create additional assertions such as \assertfalse, \assertcat for category code checks etc. The LaTeX3 Team I understand does a lot of testing; however applying the above to a learning environment I am not too sure how feasible or desirable it is to do so.