# What is the LaTeX equivalent of ConTeXt \testfeatureonce to benchmark performance

ConTeXt provides a macro \testfeatureonce to benchmark performance. The syntax is

\testfeatureonce{n}{...}


which runs the code in the second argument n times (n is assumed to be an integer) and stores the elapsed time (in seconds) in the macro \elapasedtime. The following diagnostic message is also printed on the terminal:

system > starting feature test (n=1) system > 1 feature tests done (1.353s) 

Here is a minimal ConTeXt document showing it's usage:

\starttext

\testfeatureonce{1000}
{\setbox0\hbox
{\startMPcode draw (0,0) -- (1cm, 1cm); \stopMPcode}}

\elapsedtime
\stoptext


What is the functionally equivalent macro in LaTeX used for benchmarking performance?

For all I know, there's no such macro in the LaTeX2e base (perhaps LaTeX3 has something equivalent?). But we can look up the definition of \testfeatureonce in the ConTeXt source code. The relevant definitions are in the syst-aux module.

That implementation makes use of the eTeX primitives \pdfresettimer and \pdfelapsedtime to reset an internal timer and get the elapsed time since the last reset, respectively. I didn't find a proper source for this, but it seems 65536 equals one second in the value returned by \pdfelapsedtime.

A reimplementation in pure LaTeX code might look like this:

\documentclass{article}
\usepackage{tikz}

\makeatletter
\let\resettimer=\pdfresettimer
\let\elapsedtime=\pdfelapsedtime

\newcount\c@syst@helpers@test@feature@n
\newcount\c@syst@helpers@test@feature@m

\newcommand\testfeature[2]{%
\c@syst@helpers@test@feature@m=#1\relax
\def\syst@helpers@test@feature@step{%
\ifnum\c@syst@helpers@test@feature@n>\c@syst@helpers@test@feature@m\else
#2\expandafter\syst@helpers@test@feature@step
\fi
}%
\retestfeature
}

\newcommand\retestfeature{
\bgroup
\ifcase\interactionmode \let\wait\relax \fi
\resettimer
\c@syst@helpers@test@feature@n=0\relax
\syst@helpers@test@feature@step
\wait
\egroup
}

\newcommand\testfeatureonce[2]{%
\begingroup
\let\wait\relax
\testfeature{#1}{#2}%
\endgroup
}

\makeatother

\begin{document}

\testfeatureonce{1000}
{\setbox0\hbox
{\tikz{ \draw (0,0) -- (1cm, 1cm); }}}

\the\elapsedtime

\end{document}

• Thanks! I had looked at that definition but didn't realize that etex provides the primitives \pdfresettimer and \pdfelapsedtime. – Aditya Oct 23 '18 at 0:59
• 65536 (2^16) is the number of scaled points in a point (pt) and the number of "scaled seconds" in a second. TeX represents decimal numbers as "scaled integers", this this value. Nice though, I didn't think of looking ConTeXt's definition :) +1 – Phelype Oleinik Oct 23 '18 at 3:07
• @aditya They are pdfTeX primitives also available in pTeX and upTeX. I have an open request to add them to XeTeX. – Joseph Wright Oct 26 '18 at 7:53

With expl3 you can use \tex_resettimer:D and \tex_elapsedtime:D, which exist for both pdfTeX and since TL 2019 for XeTeX (they are the primitives \(pdf)resettimer and \(pdf)elapsedtime renamed). For LuaTeX you can use the Lua functions from l3kernel. Most of this is a rudimentary version of what's in the l3benchmark package (see Joseph's answer), released shortly after this question was asked.

\documentclass{article}
\usepackage{xparse}
\usepackage{tikz}

\ExplSyntaxOn
\sys_if_engine_luatex:TF
{
\cs_new:Npn \__tfo_start_timer:  { \lua_now:n { l3kernel.resettimer() } }
\cs_new:Npn \__tfo_elapsed_time: { \lua_now:n { l3kernel.elapsedtime() } }
}
{
\cs_new_eq:NN \__tfo_start_timer: \tex_resettimer:D
\cs_new_eq:NN \__tfo_elapsed_time: \tex_elapsedtime:D
}
\cs_new:Npn \__tfo_output:n #1
{
\iow_term:x
{
>~#1~feature~tests~done~
(\fp_eval:n { round ( \__tfo_elapsed_time: / 65536 , 3 ) }s)
}
}
\cs_new:Npn \__tfo_run_n_times:nn #1 #2
{
\if_int_compare:w #1 > 0 \scan_stop:
\exp_args:No \__tfo_run_n_times:nnw { \int_value:w \__int_eval:w #1 - 1 } { #2 }
\fi:
}
\cs_new:Npn \__tfo_run_n_times:nnw #1 #2 \fi:
{
\fi:
#2
\__tfo_run_n_times:nn { #1 } { #2 }
}
\NewDocumentCommand\TestFeatureOnce
{ m m }
{
\__tfo_start_timer:
\__tfo_run_n_times:nn { #1 } { #2 }
\__tfo_output:n { #1 }
}
\ExplSyntaxOff

\begin{document}

\TestFeatureOnce{10000}
{\setbox0\hbox
{\tikz \draw (0,0) -- (1cm, 1cm);}}

\end{document}


Edit: Bug fix. A large number of tests (~5000) would exceed TeX's input stack size by piling up tons of \fi:s, so I added a helper macro to make sure everything is executed outside the \if...\fi so that an arbitrary number of runs is possible.

With jfbu's idea to convert scaled seconds, one can even do it in plain (pdf, Xe, or Lua)TeX:

\input tikz.tex

\begingroup% pdfTeX
\expandafter\ifx\csname pdfresettimer\endcsname\relax
\else
\global\let\resettimer\pdfresettimer
\global\let\elapsedtime\pdfelapsedtime
\fi
\endgroup
\begingroup% LuaTeX
\expandafter\ifx\csname directlua\endcsname\relax
\else
\gdef\resettimer{\directlua{pdfelapsedtimer_basetime = os.clock()}}
\gdef\elapsedtime{\directlua{tex.print((os.clock()-pdfelapsedtimer_basetime)*65536)}}
\fi
\endgroup
% For XeTeX the primitives are already called \resettimer and \elapsedtime
\catcode@=11
% Stolen from latex.ltx
\begingroup
\catcodeP=12
\catcodeT=12
\lowercase{%
\def\x{\def\rem@pt##1.##2PT{##1\ifnum##2>\z@.##2\fi}}}%
\expandafter\endgroup\x
\def\strip@pt{\expandafter\rem@pt\the}%
%
\let\TFO@start@timer\resettimer
\let\TFO@elapsed@time\elapsedtime
\def\TFO@output#1{%
\immediate\write17{%
> #1 feature tests done (\strip@pt\dimexpr\TFO@elapsed@time sp\relax s)
}%
}
\long\def\TFO@runNtimes@nn#1#2{%
\ifnum#1>0\relax
\expandafter\TFO@runNtimes@nnw\expandafter{\number\numexpr#1-1}{#2}%
\fi
}
\long\def\TFO@runNtimes@nnw#1#2\fi{%
\fi
#2%
\TFO@runNtimes@nn{#1}{#2}%
}
\long\def\TestFeatureOnce#1#2{%
\TFO@start@timer
\TFO@runNtimes@nn{#1}{#2}%
\TFO@output{#1}%
}
\catcode@=12

\TestFeatureOnce{1000}
{\setbox0\hbox
{\tikz \draw (0,0) -- (1cm, 1cm);}}

\bye

• Thanks. This is useful and a good introduction to Expl syntax as well. – Aditya Oct 23 '18 at 11:24
• Glad you liked :) I used expl3 mainly to use l3fp to convert the scaled seconds to seconds. The code can be translated into plain LaTeX quite easily. P.S.: Fixed a bug in the code :P – Phelype Oleinik Oct 23 '18 at 16:44
• @PhelypeOleinik to convert you only need \the\dimexpr <number>sp\relax which will give in seconds (indicated by pt...). Precise enough if your benchmark test take of the order of 1/10th to a few seconds; – user4686 Oct 23 '18 at 16:46

Code using the new l3benchmark package (released to CTAN 2018-10-26):

\documentclass{article}
\usepackage{l3benchmark}
\ExplSyntaxOn
\cs_new_protected:Npn \testfeatureonce #1#2
{
\benchmark_once:n
{ \prg_replicate:nn {#1} {#2} }
}
\ExplSyntaxOff
\usepackage{tikz}

\begin{document}
\testfeatureonce{1000}
{\setbox0\hbox
{\begin{tikzpicture} draw (0,0) -- (1cm, 1cm); \end{tikzpicture}}}
\end{document}


The benchmark code can also auto-determine how many cycles to do:

\documentclass{article}
\usepackage{l3benchmark}
\ExplSyntaxOn
\cs_new_protected:Npn \testfeatureonce #1#2
{ \benchmark:n {#2} }
\ExplSyntaxOff
\usepackage{tikz}

\begin{document}
\testfeatureonce{1000}
{\setbox0\hbox
{\begin{tikzpicture} draw (0,0) -- (1cm, 1cm); \end{tikzpicture}}}
\end{document}


(The code itself is of course much the same as the other answers here.)

• Thanks. What is the algorithm to determine the number of cycles? – Aditya Oct 23 '18 at 22:10
• @Aditya You set a target time (default: 1s). The system then starts by doing one run of the user code, and timing it. If that takes over half the target, we are done. If not, we run the user code 4 times, and see the timing. That keeps happening until we get to at least half of the target time. Then we divide the time required by the number of loops, to give an shortest time to execute the user code. – Joseph Wright Oct 24 '18 at 6:27

A primitive benchmarking utility:

\documentclass{article}

\usepackage{xintkernel}

\makeatletter
\def\testfeatureonce#1#2{\pdfresettimer
\romannumeral\xintreplicate{#1}{#2}%
\edef\theelapsedtime{\strip@pt\dimexpr\pdfelapsedtime sp\relax s}%
% \theelapsedtime % commented-out
}
\makeatother

\usepackage{tikz}

\begin{document}

\testfeatureonce{1000}
{\setbox0\hbox
{\tikz{ \draw (0,0) -- (1cm, 1cm); }}}

\typeout{Previous test took \theelapsedtime}

\end{document}


The result is stored (but after conversion to seconds) for being re-usable.

....
(/usr/local/texlive/2018/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg))
Previous test took 0.92303s
[1{/usr/local/texlive/2018/texmf-var/fonts/map/pdftex/updmap/pdftex.map}]
....


The package xintkernel is very low weight package (provides also \xintUniformDeviate) which has nothing of the computations facilities of xint itself.

The \xintreplicate is basically the same code (plagiarized) originally in l3kernel.

For more advanced benchmarking see l3benchmark.

The above does not compute the overhead from the expansion of \xintreplicate (which here would be a tiny fraction of the tikz things). If you want 1000000 replications better to nest two \romannumeral\xintreplicate{1000}.

• Thanks. Why does \xintreplicate need to be preferred by \romannumeral? – Aditya Oct 23 '18 at 18:36
• @Aditya well, in xint, most macros have a capitalized version \xintFoo and lower case version \xintfoo with \xintFoo expanding to \romannumeral0\xintfoo, allowing to nest macros and maintain complete expandability in two steps (using \def\xintBar#1#2{\xintFoo{...}{...}} would mean three steps.) In the case of \xintreplicate I did not define the user level \xintReplicate. And it uses \romannumeral because it inserts a \z@ when done to stop it, rather than most xint macros which trigger expansion by \romannumeral0 and stop it by inserting a space token at the end. – user4686 Oct 23 '18 at 21:14