Recommended means of testing packages while writing them

Question

What is the recommended way of writing a package in the .dtx format? Do package authors usually just write the .dtx first, or use some kind of tool to create a .dtx from the .sty files? I know both approaches are valid, but what's the best way to get started? I'm currently trying to make sense of the available documentation, but like LaTeX in general this feels like it has a learning curve not unlike a brick wall.

Can LaTeX load .dtx files as a substitute for .sty files, or must they be preprocessed first?

Prior packages that I've written were just collections of \usepackage and various convenience macros, but now I'm trying to write something more useful that I'd like to distribute at some point, and contribute back to the community.

Answer 1

Disclaimer: I am a package developer, but solely for internal use (me/my institution; 9 packages and 3 classes so far). I have not uploaded a package to CTAN yet. Hence this is not the advice a more experienced LaTeX developer might give.

I usually structure the development of my packages (which are dtx files at the end, using l3build as working environment) into three stages:

Stage 1: Getting the concept right

This stage involves trial and error. I set up a test document with the syntax I want and make "dummy commands" for all undefined control sequences, which effectively do nothing at all.

Then I'm setting up a basic package as a .sty file (using expl3 syntax). Now I'm trying to implement the essential macros which are needed as well as package options. For each macro implemented I remove my dummy macro.

This step lasts until I think that the basic functionality of my package is implemented. While implementing you are easily able to test.

Stage 2: Documenting your progress

In this step I'm transferring the code into dtx format (using the l3doc functionality). This step is very tedious, but important. Basically you add the comment structure to your code. At this point I also make the package a git project for version control.

Do not forget to write down something about the basic concept of your package and a simple roadmap, especially if you know you will have pauses in development.

While documenting I utilize l3build to build the documentation for me. Usually I also include my package in my dtx documentation (which may contain examples), so severe issues will be recognized very fast. l3build lets you also create test files, which are a good option for more complicated packages.¹

Stage 3: Further development, tests and releases

Your dtx file is ready for additional functionality and updates of existing macros. I'm altering the dtx file directly (git is my insurance if something goes wrong). As l3build also creates the package with every run, if you tell it so², you may also just copy the .sty file and test some modifications with that beforehand.

As I include the package within the dtx documentation and usually write up examples for the functions of my package I will also notice some issues just while extending the package.

If you think you are at a point the functionality works, turn your dtx file into a full documentation (for the user) and think of some test cases (what can the user do wrong). Test the package. I usually take my test file from step 1 as a starter.

Then you can use l3build to cmdcheck your dtx and make a ctan build. You may create a git tag at this point (if you're using git).

---

¹ It is slightly harder to do this for classes as you probably won't write your documentation with your class, but you may want to include some example file into the documentation which is compiled every doc build just for testing.
² You may basically include the .ins file into the .dtx file and get the package created at every run. There are examples out there how to do it.

Answer 2

The first thing to remember is that you do not have to use the .dtx format to include comments in package code: you can just use a .sty. Assuming you do want to use .dtx to allow typesetting the code, etc., then you still may be able to use the .dtx directly. That is true if there is only a single 'target' in the source file: just comments and the 'live' code. However, most .dtx files include the 'driver' (to allow typesetting the code), and so effectively at least two targets. At this stage you have to extract the .sty to be able to use it with LaTeX.

If we take this common situation, you can test by having a simple .tex file in the same place as the .dtx and using shell escape to enable a first line such as

\immediate\write18{tex mypkg.dtx}
\documentclass{article}
\usepackage{mypkg}
% Tests

That of course has a non-zero security implication and tends to be a bit 'fragile': ideally you want a sandbox for testing that keeps extracted code separate from everything else.

The obvious approach is to script extraction, perhaps installation in the local tree and testing. There are several scripts which can do that, but unsurprisingly I'd point to l3build from the LaTeX3 Project. This is Lua-based script which allows fully automated testing of packaged code: depending on what you want, you might just unpack and test with a simple .tex file, or might create one or more automated test input/output sets (.lvt/.tlg files).

In terms of creating .dtx files, there are scripts to 'fill in the blanks' but in the end the point of the format is to have useful comments/description both for documentation and in the code. That can only be done by you actually writing the material!