# What are the exact semantics of \detokenize?

How does \detokenize work?

The e-TeX Short Reference Manual states:

When followed by a <general text>, expands to yield a sequence of character tokens of \catcode 10 (space) or 12 (other) corresponding to a decomposition of the tokens of the <balanced text> of the unexpanded <general text>; c.f. \showtokens. The effect is rather as if \scantokens applied to the <general text> within a régime in which only \catcodes 10 and 12 existed. Note that in order to preserve the boundaries between control words and any following letter, a space is yielded after each control word including the last.

What does that mean in plain English? I want to understand this snippet of code:

\def\task#1{\@task#1::\@nil}
\if\relax\detokenize{#2}\relax
...


I think that \task is defined as a macro taking one argument which calls \@task. I don't know what \@nil means -- it is probably used to end the macro call. I also don't know what \relax and \detokenize do, but detokenize takes the second part of the argument for the task macro (the parts are separated with semicolons).

-

\task takes one argument, passing it to \@task which is defined in such a way that its arguments are delimited; if the call is

\@task xyz:AB:cde:u\@nil


the first argument is xyz, the second is AB and the third is cde:u. Here \@nil doesn't mean anything, it's just required by the syntax of \@task and TeX throws it away.

\relax is a primitive of TeX, its function is "do nothing". The test

\if\relax\detokenize{#2}\relax


is a safe way to determine if the argument #2 is empty. If it is, \detokenize{#2} expands to nothing, so \if compares the tokens \relax and \relax, which are indeed equal, so the "true" branch is followed, which starts immediately after the second \relax, up to and excluding \else. If #2 is not empty, say it's 30, \if will compare \relax with 3 which are different, so the "false" branch is followed, which starts after \else up to and excluding \fi.[1]

The similar construct \if\relax#2\relax does not work in all cases, because "all control sequences are equal as far as \if is concerned" [2]. It would not work if #2 was \relax (or any other control sequence, possibly followed by other tokens)! So we use \detokenize that, as explained also by Joseph, splits everything into a string. So, even in the weird case that #2 is \relax, \if would compare the token \relax with the character \, which are different.

[1] This is not strictly true, but it's an approximation of the truth sufficient for the purpose of this description.

[2] TeX by Topic (section 13.2.1)

-
Thank you again! I think that I understand it now. There are two more points that are not that clear to me: 1. Why are arguments for \@task split like that? xyz+AB+cde or xyz+AB:cde+u or xyz+AB+cde:u all seem to satisfy <part>:<part>:<part>. 2. Will \detokenize split \relax into \  r e l a x? –  ipavlic Jun 6 '11 at 14:50
@ipavlic 1. TeX takes always the shortest token list that satisfies the requests (as long as the braces are balanced) so the first argument is anything up to the first colon, the second is anything from the first to the second colon and the third is anything from the second colon to \@nil. 2. Yes. –  egreg Jun 6 '11 at 14:57
Doesn't then the third argument has to be cde:u instead of u as written in the answer? This should not be nitpicking, I am really not sure enough to edit it. If I am wrong, please explain why. –  Patrick Häcker Oct 9 '12 at 17:19
@MMM You're right. –  egreg Oct 9 '12 at 17:28

The \detokenize primitive converts all of its input into category code 12 ('other') tokens, except for spaces which stay as category code 10 ('space'). It is used to convert tokens into strings. For example, if we have

\detokenize{\foo}


then \foo would be a control word, i.e. one token, which is converted into 5 string characters (\, f, o, o, ), all which are 'other' characters other than the space, which is a 'space'. Note the additional space at the end here: \detokenize adds a space after each 'control word' (see below).

I'll take your example one step at a time. First, \@nil is being used as a delimiter here. The definition of \@task means that the argument must contain :, then : again and finally \@nil. This is used to 'tidy up' awkward input

\task{stuff:more stuff}


will be turned into

 \@task stuff:more stuff::\@nil


which means #1 is stuff, #2 is more stuff and #3 is :. Thus it ensures that everything is 'used up': this is a common requirement.

To see \detokenize in action, try something like

\documentclass{article}
\usepackage[T1]{fontenc}
\edef\test{\detokenize{\test}}
\show\test
\begin{document}
\detokenize{\test}
\end{document}


A bit more on the addition of spaces by \detokenize, and indeed more generally. Whenever TeX writes something that can be one or more tokens as a 'string', it always inserts a space after each 'control word' (escape character followed by one or more 'letters') to avoid confusion. For example, with

\catcode\X=12 % 'Other'
\def\foo{\barX}
\meaning\foo


you get

macro:-> \bar X


where the space is not there in the original but tells us that \bar is a control sequence followed by X, rather than printing just \barX, which looks like it's a different thing. This makes sense as TeX skips spaces after control words, so the two versions are equivalent as far as it is concerned.

This 'insert spaces' behaviour is respected by \detokenize, meaning that you get a space after each control word. The only place TeX doesn't insert these spaces is with \string, which can only be used on a single token and so in places where there is no possible 'confusion'.

-
Let me see if I understand it correctly. #1 from task expands to stuff:more stuff in the command \@task stuff:more stuff::\@nil. The purpose of the added :: is to enable input without semicolons, e.g. \task{1}. Now the semicolons will be concatenated to 1 to form 1:: which will satisfy the two-semicolons requirement of \@task. –  ipavlic Jun 6 '11 at 14:43
@ipavlic: #1 is never expanded, rather it is replaced by the input, but the result is as you say \@task stuff:more stuff::\@nil is executed. You are also right about the :: part - without this, you'd be in trouble with non-conforming input. –  Joseph Wright Jun 6 '11 at 14:47
@ipavlic: \detokenize leaves the result in the input stream. Thus you end up with \detokenize{foo} => foo` wherever it was used. In the current case, this is as part of a test, which will throw away any excess tokens. On the other hand, if you use it in a typesetting context then the material will just be typeset. See my edit –  Joseph Wright Jun 6 '11 at 14:49
@Joseph: I guess that the word expand has a special meaning in LaTeX, so "replace" would be more fitting. If I understand correctly, depending on the context, the commands behave a bit differently -- sometimes the parts of the content are replaced, and sometimes they are typeset "as they are". I think I am beginning to understand the concept of how LaTeX works. Thank you! –  ipavlic Jun 6 '11 at 14:57
@ipavlic: This is all TeX, not LaTeX per se. Expansion has a very precise meaning in TeX programming, and for low-level TeX questions like this one I think it's important to be careful about the terminology. –  Joseph Wright Jun 6 '11 at 15:02