# Question about \maketoks macros from cwebmac.tex

Consider the following code from cwebmac.tex:

{\setbox0=\hbox{\toksA={1.}\toksB={}\maketoks}\the\toksA}


What does \maketoks macros do and how does it work?

The following is a complete example (reduced from cwebmac.tex) for pdftex:

\newtoks\toksA \newtoks\toksB \newtoks\toksC \newtoks\toksD

\newcount\countB \newcount\countC

\def\makenote{%
\toksC={}%
\global\countC=0
}

\def\poptoks#1#2|ENDTOKS|{\let\first=#1\toksD={#1}%
\ifcat\noexpand\first0\countB=#1\else\countB=0\fi\toksA={#2}}

\def\maketoks{\expandafter\poptoks\the\toksA|ENDTOKS|%
\ifnum\countB>9 \countB=0 \fi
\ifnum\countB<0
\ifnum0=\countC\else\makenote\fi
\ifx\first.%
\let\next=\maketoksdone
\else
\let\next=\maketoks
\fi
\else
\global\countC=1
\let\next=\maketoks
\fi
\next
}

\def\maketoksdone{\edef\st{\global\noexpand\toksA={\the\toksB}}\st}

{\setbox0=\hbox{\toksA={1.}\toksB={}\maketoks}\the\toksA}
\bye


EDIT

Consider the following test.w (I disable \startsection intentionally not to set destination marks):

\nosecs
\let\startsection\empty
@* Example.
@c
@<Something@>;
@ @<Something@>=
void main (void)


Run cweave test.w, then process test.tex

1) via pdftex

pdftex test.tex


2) via dvipdfmx

tex '\let\pdf+ \input test.tex'
dvipdfmx test.dvi


We get the following errors:

1)

pdfTeX warning (dest): num1 has been referenced but does not exist, replaced by a fixed one
pdfTeX warning (dest): num2 has been referenced but does not exist, replaced by a fixed one


2)

dvipdfmx:warning: PDF destination "ii" not defined.
dvipdfmx:warning: PDF destination "i" not defined.


The conclusion is that this romannumeral thing is simply to set destination mark in dvipdfmx. Maybe it may be replaced simply to num #1 like for pdftex and then all this romannumeral stuff will not be necessary?

Macro \maketoks parses a token list variable \toksA and transforms its contents, using temporary \toksD, and also \toksB which accumulates the transformed material before delivery into final \toksA (\toksB may contain previous stuff too). Also in use a \toksC whose rôle is described next.

Notice that to analyse one needs to know catcodes of |ENDTOKS|, I assume | has catcode other and the letters catcode letter. Well no it seems that the loop must end before hitting |ENDTOKS|, hence it is assumed that the exit point is hit, which means (see below) that a dot must be found at some point.

A loop is entered. The encountered tokens are examined one by one. Notice though that the code is not a token per token analyzer per se, things are grabbed as macro parameter #1 hence it seems to be assumed the encountered material has no braced material for example, and space tokens will not be seen (and are pruned).

1. if the token is a digit of catcode 12, this digit is appended to \toksC and \countC flag is set to numeric value 1.

2. if the token is not of catcode 12, or is of catcode 12 but is not a digit:

a. if a digit was already previously encountered, then that (continuous series of) digit(s) is appended to \toksB and the \toksC is reset to empty and \countC reset to zero, and one continues as below.

b. else if the token is a dot . then the loop is exited, and the accumulated data put into \toksA (without the dot). Remaining stuff is discarded. Notice that there is a stray space token here (no % after \ifx\first.), but apparently the whole thing is executed inside a temporary box, hence does not influence typesetting.

c. else the token is appended to \toksB. It it was a comma, an extra space is appended. Here also the code has a spurious space token (no % after \addtokens\toksB{\the\toksD}), but apparently the whole thing is destined to be executed inside a temporary box, hence those spaces do not matter. Then the loop is continued.

In short the effect is that the part of \toksA until the first encountered dot is kept, (possibly the dot was even hidden in a {.}), the rest is discarded, and spaces are added after commas. I don't quite get it regarding digits, why are they accumulated in \toksC and not directly in \toksB, because in the end their fate seems no different from other tokens (which I assume will mainly be characters).

Igor has clarified in comments that the posted code differed a bit from the original one in cwebmac.tex. The original code indeed has special treatment for the parsed continuous series of digits, particularly it seems to create PDF links, with different syntax if compilation goes via pdftex or for example tex+dvipdfmx.

% standard macros for CWEB listings (in addition to plain.tex)
% Version 3.68 --- January 2016


One finds:

\def\makenote{\addtokens\toksB


and also:

\ifpdftex
attr{/Border [0 0 0]} goto num #1 \BlueGreen #1\Black\pdfendlink}}
\special{pdf: ec}}\special{pdf: ann width \thewidth height \theheight
depth \thedepth << /Type /Annot /Subtype /Link
/Border [0 0 0] /A << /S /GoTo /D (#2) >> >>}\box0\relax}\fi


The pdftex case uses \pdfannotlink=\pdfstartlink from PDFTeX and uses goto num and not the #2 which has the TeX \romannumeral whereas the dvipdfmx case uses the #2 for a conversion to roman numerals.

Elsewhere (in the definition of \startsection) one finds examples of how destination names are created:

  \ifpdftex\smash{\raise\baselineskip\hbox to0pt{%
%     \let\*=\empty\pdfdest num \secstar fith}} % bad space in versions < 3.68
\let\*=\empty\pdfdest num \secstar fith}}% changed in version 3.68
\else\ifpdf\smash{\raise\baselineskip\hbox to0pt{%
\let\*=\empty\special{%
pdf: dest (\romannumeral\secstar) [ @thispage /FitH @ypos ]}}}\fi\fi


All this to confirm that the parsing of digits done by the original \maketoks was indeed not gratuitous ;-).

I am not familiar (this is an euphemism) with the PDF specs, I can only conjecture that one can not use target identifiers starting with a digit or consisting of digits, and that either the num keyword, or the use of \romannumeral is a work-around. This is pure speculation.

• "...digits are transformed via \romannumeral..." - the whole {\romannumeral...} is argument #2, which is totally ignored in \def\pdflink in pdftex – Igor Liferenko Mar 29 '16 at 9:45
• {\setbox0=\hbox{\toksA={1.}\toksB={}\maketoks}\the\toksA} in my question is equivalent to {\pdfnote#1.} in cwebmac.tex (I have reduced not essential parts to make the example code simpler) – Igor Liferenko Mar 29 '16 at 9:46
• \pdflink was omitted - for the same reason of making the code simpler - it is not essential (it may be simply replaced with the number - the resulting pdf will look the same) – Igor Liferenko Mar 29 '16 at 9:48
• about your first comment, you are right, but there is also a \def\pdflink which does not ignore the #2, if conditional \ifpdftex is false (I guess for dvipdfm/dvipdfmx or possibly for dvips then ps2pdf, this is beyond my competence). Sorry in advance but I can't correct my answer until possibly tomorrow for lack of time. – user4686 Mar 29 '16 at 11:56
• Indeed, the second \def\pdflink (which uses #2) is for dvipdfmx. You are right: digits are accumulated in \toksC specifically to store the number, which will be converted to roman numeral. BTW, do you know what for we use /D (<roman numeral>)` in dvipdfmx's specials for pdf outlines? Is it some kind of depth level? – Igor Liferenko Mar 30 '16 at 2:46