What are the exact semantics of \detokenize?

Question

How does \detokenize work?

The e-TeX Short Reference Manual states:

When followed by a <general text>, expands to yield a sequence of character tokens of \catcode 10 (space) or 12 (other) corresponding to a decomposition of the tokens of the <balanced text> of the unexpanded <general text>; c.f. \showtokens. The effect is rather as if \scantokens applied to the <general text> within a régime in which only \catcodes 10 and 12 existed. Note that in order to preserve the boundaries between control words and any following letter, a space is yielded after each control word including the last.

What does that mean in plain English? I want to understand this snippet of code:

\def\task#1{\@task#1::\@nil}
\def\@task#1:#2:#3\@nil{%
  \if\relax\detokenize{#2}\relax
  ...

I think that \task is defined as a macro taking one argument which calls \@task. I don't know what \@nil means -- it is probably used to end the macro call. I also don't know what \relax and \detokenize do, but detokenize takes the second part of the argument for the task macro (the parts are separated with semicolons).

Answer 1

\task takes one argument, passing it to \@task which is defined in such a way that its arguments are delimited; if the call is

\@task xyz:AB:cde:u\@nil

the first argument is xyz, the second is AB and the third is cde:u. Here \@nil doesn't mean anything, it's just required by the syntax of \@task and TeX throws it away.

\relax is a primitive of TeX, its function is "do nothing". The test

\if\relax\detokenize{#2}\relax

is a safe way to determine if the argument #2 is empty. If it is, \detokenize{#2} expands to nothing, so \if compares the tokens \relax and \relax, which are indeed equal, so the "true" branch is followed, which starts immediately after the second \relax, up to and excluding \else. If #2 is not empty, say it's 30, \if will compare \relax with 3 which are different, so the "false" branch is followed, which starts after \else up to and excluding \fi.[1]

The similar construct \if\relax#2\relax does not work in all cases, because "all control sequences are equal as far as \if is concerned" [2]. It would not work if #2 was \relax (or any other control sequence, possibly followed by other tokens)! So we use \detokenize that, as explained also by Joseph, splits everything into a string. So, even in the weird case that #2 is \relax, \if would compare the token \relax with the character \, which are different.

---

[1] This is not strictly true, but it's an approximation of the truth sufficient for the purpose of this description.

[2] TeX by Topic (section 13.2.1)

Answer 2

The \detokenize primitive converts all of its input into category code 12 ('other') tokens, except for spaces which stay as category code 10 ('space'). It is used to convert tokens into strings. For example, if we have

\detokenize{\foo}

then \foo would be a control word, i.e. one token, which is converted into 5 string characters (\, f, o, o, ), all which are 'other' characters other than the space, which is a 'space'. Note the additional space at the end here: \detokenize adds a space after each 'control word' (see below).

I'll take your example one step at a time. First, \@nil is being used as a delimiter here. The definition of \@task means that the argument must contain :, then : again and finally \@nil. This is used to 'tidy up' awkward input

\task{stuff:more stuff}

will be turned into

 \@task stuff:more stuff::\@nil

which means #1 is stuff, #2 is more stuff and #3 is :. Thus it ensures that everything is 'used up': this is a common requirement.

---

To see \detokenize in action, try something like

\documentclass{article}
\usepackage[T1]{fontenc}
\edef\test{\detokenize{\test}}
\show\test
\begin{document}
\detokenize{\test}
\end{document}

---

A bit more on the addition of spaces by \detokenize, and indeed more generally. Whenever TeX writes something that can be one or more tokens as a 'string', it always inserts a space after each 'control word' (escape character followed by one or more 'letters') to avoid confusion. For example, with

\catcode`\X=12 % 'Other'
\def\foo{\barX}
\meaning\foo

you get

macro:-> \bar X

where the space is not there in the original but tells us that \bar is a control sequence followed by X, rather than printing just \barX, which looks like it's a different thing. This makes sense as TeX skips spaces after control words, so the two versions are equivalent as far as it is concerned.

This 'insert spaces' behaviour is respected by \detokenize, meaning that you get a space after each control word. The only place TeX doesn't insert these spaces is with \string, which can only be used on a single token and so in places where there is no possible 'confusion'.