(How) Can I write a .dtx file without having to comment out *everything*?

I have a reasonable understanding now of how .dtx files work. But it's a great pain having to put a % character before every line of documentation. Also, depending on which editor you're using, it basically means: no syntax highlighting.

This problem has been noted before. But the only solutions offered were about editor settings. Plus, even if we forget about syntax highlighting, the % thing is just... inconvenient.

So I'm looking for a way to dispense with all the commenting. Basically, I'd like to delimit documentation code using guard modifiers. Something like this:

%<*documentation>
I'm going to introduce a cool new \LaTeX macro:
\begin{macrocode}
%</documentation>

\newcommand{\CoolNewMacro}{...}

%<*documentation>
\end{macrocode}
Cool, huh?
%</documentation>


I think I could figure out how to do it by first extracting the documentation code into a separate file. But... do I need to? Is there a better way?

-
I don't think it makes sense to use an editor which can not even do that. –  Stephan Lehmke Nov 23 '12 at 7:14
@StephanLehmke I'd rather put it like this: I don't think it makes sense to use a language which requires you to prefix so many lines with a comment symbol. ;-) --- Anyway, I'm using Eclipse with the TeXLipse plugin. I pretty much use Eclipse for everything, and for LaTeX it has some great features (such as synctex). This just happens not to be one of them. –  mhelvens Nov 23 '12 at 9:23
@StephanLehmke I guess you're talking about Javadoc. Two important differences: (1) it still allows you to delimit documentation with /** */ thus not requiring you to comment every line. (2) There you write down only the bare minimum of information about defined classes and methods and documentation is automatically extracted. You don't write the manual. –  mhelvens Nov 23 '12 at 9:48
@mhelvens "You don't write the manual." You don't need to do that in TeX either. The .drv usually mostly contains \DocInput{\jobname.dtx} and the whole doc is in the .dtx, but that's just convention. You can as well put the whole manual in the drv as a plain normal LaTeX document and input only the documented sources with \DocInput{\jobname.dtx}. –  Stephan Lehmke Nov 23 '12 at 9:54
By now the "doc - docstrip" workflow in the TeX community is a habit, and it seems that it won't change easily. I think you will fit right with coding in real literate-style instead of putting % at every line of documentation. Take a look at noweb. It's easy and useful. –  Jean Baldraque Nov 24 '12 at 9:52

If you look at the history of doc, then it is understandable why the current system works as it works. Initially the idea as to write a .sty file with the comments imbedded, but in a way that it would directly be processable by LaTeX. And that required putting every bit of documentation behind % signs.

Only later docstrip appeared (to strip the comments because back then processing all those unnecessary comment lines took noticable extra time). Even later the <guards> appeared and with them none-sequencial generation of target files. At that time the name of the files changed to .dtx and one could then have changed the documentation part to work without % --- but that didn't happen.

Having said this, one can with a little modification run doc in that form, e.g.,

%<*documentation>
\documentclass{article}

\usepackage{doc}

\makeatletter
\def\inlinecode{\macro@code
\frenchspacing \@vobeyspaces
\xinline@code}
\let\endinlinecode\endmacrocode

\begingroup
\catcode\|=\z@ \catcode$=\@ne \catcode$=\tw@
\catcode\{=12 \catcode\}=12
\catcode\\=\active
|gdef|xinline@code#1\end{inlinecode}[#1|end[inlinecode]]
|endgroup
\makeatother

\begin{document}
\noindent some text
\begin{inlinecode}
%</documentation>
\def\foo{bar}
%<*documentation>
\end{inlinecode}
more text
\end{document}
%</documentation>


To the processable code out of this file one would need a docstrip install file like

\input docstrip

\generate{\file{x.sty}{\from{how-can-i-write-a-dtx-file-without-having-to-comment-out-everything.dtx}{}}}

\endbatchfile


But it isn't fully satisfying, as the documentation guards show up in the documentation and without some serious changes to docstrip and doc I don't see a way to get rid of those:

perhaps it would look a little better in print if we call the guard "code" (but then * and / are in the wrong order) --- either way it is suboptimal.

Update

If one wants to get rid of the documentation guards, one simple solution is of course to use docstrip to generate a separate documentation file as it was suggested already by the OP. To do this all one has to do is to additionally provide the line

\generate{\file{packagedoc.tex}{\from{how-can-i-write-a-dtx-file-without-having-to-comment-out-everything.dtx}{documentation}}}


in the .ins file and then run LaTeX on the resulting file. However, to fully utialize the features of doc, e.g., the code indexing or the code line numbering it is imortant to use the macrocode environment. As this environment uses the special syntax with % \end{macrocode} it can't be used directly at least not if one wants to avoid putting the % in. Therefore the code above defining the inlinecode environment as an alternative is essential.

Perhaps that bit of code should be added to doc to allow for this approach.

-
Hi! Your catcode-intensive solution is very interesting, but I find it a bit hard to follow (as I do all catcode-intensive TeX code). -<###>- And I have to say, the fact that those tags would remain visible in the documentation makes this solution... unacceptable. :-) I think the answer I just proposed myself is a bit more workable. -<###>- Nonetheless, I'll have some fun puzzling out your catcodes later. Cheers! –  mhelvens Nov 24 '12 at 19:18
@mhelvens with a bit more effort it is possible to suppress the documentation guards and leave just an empty line there. Getting rid of that line is more difficult as with \CodelineIndex the line would have a number attached. The catcode tricks are the same as used in doc.dtx, so reading its documentation should help :-). As to the visible guards: your solution shows them too, so why do you think it is more workable? –  Frank Mittelbach Nov 25 '12 at 8:42

Ok, so maybe it's not possible to do this elegantly using the 'standard' way of running the .dtx file through LaTeX to get the docs. But I have an alternative solution here which seems just as nice.

With this setup, there is only one file (the '.ins file' is included). You run it through LaTeX and it generates package.sty and package.tex. To get the actual documentation you still need to run package.tex itself through LaTeX. (A sacrifice I am willing to make.)

Here is a minimal working example (and a reasonable skeleton for anyone wanting to do the same):

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%% \CheckSum{3}
%%
%% \CharacterTable
%%  {Upper-case    \A\B\C\D\E\F\G\H\I\J\K\L\M\N\O\P\Q\R\S\T\U\V\W\X\Y\Z
%%   Lower-case    \a\b\c\d\e\f\g\h\i\j\k\l\m\n\o\p\q\r\s\t\u\v\w\x\y\z
%%   Digits        \0\1\2\3\4\5\6\7\8\9
%%   Exclamation   \!     Double quote  \"     Hash (number) \#
Acute accent  \'     Left paren    $$Right paren$$
Asterisk      \*     Plus          \+     Comma         \,
Minus         \-     Point         \.     Solidus       \/
Colon         \:     Semicolon     \;     Less than     \<
Equals        \=     Greater than  \>     Question mark \?
Commercial at \@     Left bracket  $Backslash \\ Right bracket$     Circumflex    \^     Underscore    \_
Grave accent  \     Left brace    \{     Vertical bar  \|
Right brace   \}     Tilde         \~}

\CheckSum{2}

\Finale
%%
%% End of file


The actual situation in my dtx file is more complicated, I simplified things here to get that skeleton. I agree that having all those % is indeed a problem, although with emacs there is a mode which does the LaTeX syntax highlighting nevertheless. So I was also interested in finding a way not to have all those %'s.

Oups, I forgot to say that the ins stuff is purely optional. It is only to produce an .ins file, but it is not used, as the .sty file itself is directly output.

-
Ah, I see. The trick here, then, is to manually call \MakePercentIgnore before starting the 'implementation' section. So you have most of the documentation without the %s, but you still need them for that last section. Correct? Interesting solution. –  mhelvens Nov 24 '12 at 21:47
@mhelvens: Yes, the commenting of the source code is still done with the %, in the standard manner. But the user manual can be entirely written without the %. The important thing is when I realized I should forget entirely about using \DocInput!! –  jfbu Nov 24 '12 at 21:50
Yes, I did the same thing in my own solution, but took it a bit further. I'm not commenting out lines of code anywhere. Whether or not that's better can be debated. But I suppose you could say that my solution is more purely literate programming, as the program is 'submissive' to the documentation. That is to say: it is merely part of the story. In your solution, the documentation still takes a back seat, so to say. But I suppose this is bordering on the philosophical. –  mhelvens Nov 24 '12 at 22:40
@mhelvens: interesting! I didn't think about going as far as you did, simply because Emacs handles beautifully the % at the start of lines, it is just as if they were not there for syntax highlighting and automatized filling. However in the user manual part I may have to copy paste to plain .tex file and then the % are slightly a nuisance. So I was mainly keen on getting rid of them in this user manual part, before the commented source code. –  jfbu Nov 29 '12 at 7:26
@mhelvens: and of course no line of code is commented out in the implementation part, only the lines of commenting are commented out, in the standard macrocode manner. –  jfbu Nov 29 '12 at 7:27