How does \detokenize work?

The e-TeX Short Reference Manual states:

When followed by a <general text>, expands to yield a sequence of character tokens of \catcode 10 (space) or 12 (other) corresponding to a decomposition of the tokens of the <balanced text> of the unexpanded <general text>; c.f. \showtokens. The effect is rather as if \scantokens applied to the <general text> within a régime in which only \catcodes 10 and 12 existed. Note that in order to preserve the boundaries between control words and any following letter, a space is yielded after each control word including the last.

What does that mean in plain English? I want to understand this snippet of code:


I think that \task is defined as a macro taking one argument which calls \@task. I don't know what \@nil means -- it is probably used to end the macro call. I also don't know what \relax and \detokenize do, but detokenize takes the second part of the argument for the task macro (the parts are separated with semicolons).

2 Answers 2


\task takes one argument, passing it to \@task which is defined in such a way that its arguments are delimited; if the call is

\@task xyz:AB:cde:u\@nil

the first argument is xyz, the second is AB and the third is cde:u. Here \@nil doesn't mean anything, it's just required by the syntax of \@task and TeX throws it away.

\relax is a primitive of TeX, its function is "do nothing". The test


is a safe way to determine if the argument #2 is empty. If it is, \detokenize{#2} expands to nothing, so \if compares the tokens \relax and \relax, which are indeed equal, so the "true" branch is followed, which starts immediately after the second \relax, up to and excluding \else. If #2 is not empty, say it's 30, \if will compare \relax with 3 which are different, so the "false" branch is followed, which starts after \else up to and excluding \fi.[1]

The similar construct \if\relax#2\relax does not work in all cases, because "all control sequences are equal as far as \if is concerned" [2]. It would not work if #2 was \relax (or any other control sequence, possibly followed by other tokens)! So we use \detokenize that, as explained also by Joseph, splits everything into a string. So, even in the weird case that #2 is \relax, \if would compare the token \relax with the character \, which are different.

[1] This is not strictly true, but it's an approximation of the truth sufficient for the purpose of this description.

[2] TeX by Topic (section 13.2.1)

  • 1
    Thank you again! I think that I understand it now. There are two more points that are not that clear to me: 1. Why are arguments for \@task split like that? xyz+AB+cde or xyz+AB:cde+u or xyz+AB+cde:u all seem to satisfy <part>:<part>:<part>. 2. Will \detokenize split \relax into \ r e l a x?
    – ipavlic
    Jun 6, 2011 at 14:50
  • 1
    @ipavlic 1. TeX takes always the shortest token list that satisfies the requests (as long as the braces are balanced) so the first argument is anything up to the first colon, the second is anything from the first to the second colon and the third is anything from the second colon to \@nil. 2. Yes.
    – egreg
    Jun 6, 2011 at 14:57
  • Doesn't then the third argument has to be cde:u instead of u as written in the answer? This should not be nitpicking, I am really not sure enough to edit it. If I am wrong, please explain why. Oct 9, 2012 at 17:19
  • @MMM You're right.
    – egreg
    Oct 9, 2012 at 17:28

The \detokenize primitive converts all of its input into category code 12 ('other') tokens, except for spaces which stay as category code 10 ('space'). It is used to convert tokens into strings. For example, if we have


then \foo would be a control word, i.e. one token, which is converted into 5 string characters (\, f, o, o, ), all which are 'other' characters other than the space, which is a 'space'. Note the additional space at the end here: \detokenize adds a space after each 'control word' (see below).

I'll take your example one step at a time. First, \@nil is being used as a delimiter here. The definition of \@task means that the argument must contain :, then : again and finally \@nil. This is used to 'tidy up' awkward input

\task{stuff:more stuff}

will be turned into

 \@task stuff:more stuff::\@nil

which means #1 is stuff, #2 is more stuff and #3 is :. Thus it ensures that everything is 'used up': this is a common requirement.

To see \detokenize in action, try something like


A bit more on the addition of spaces by \detokenize, and indeed more generally. Whenever TeX writes something that can be one or more tokens as a 'string', it always inserts a space after each 'control word' (escape character followed by one or more 'letters') to avoid confusion. For example, with

\catcode`\X=12 % 'Other'

you get

macro:-> \bar X

where the space is not there in the original but tells us that \bar is a control sequence followed by X, rather than printing just \barX, which looks like it's a different thing. This makes sense as TeX skips spaces after control words, so the two versions are equivalent as far as it is concerned.

This 'insert spaces' behaviour is respected by \detokenize, meaning that you get a space after each control word. The only place TeX doesn't insert these spaces is with \string, which can only be used on a single token and so in places where there is no possible 'confusion'.

  • Let me see if I understand it correctly. #1 from task expands to stuff:more stuff in the command \@task stuff:more stuff::\@nil. The purpose of the added :: is to enable input without semicolons, e.g. \task{1}. Now the semicolons will be concatenated to 1 to form 1:: which will satisfy the two-semicolons requirement of \@task.
    – ipavlic
    Jun 6, 2011 at 14:43
  • @ipavlic: #1 is never expanded, rather it is replaced by the input, but the result is as you say \@task stuff:more stuff::\@nil is executed. You are also right about the :: part - without this, you'd be in trouble with non-conforming input.
    – Joseph Wright
    Jun 6, 2011 at 14:47
  • 1
    @ipavlic: \detokenize leaves the result in the input stream. Thus you end up with \detokenize{foo} => foo wherever it was used. In the current case, this is as part of a test, which will throw away any excess tokens. On the other hand, if you use it in a typesetting context then the material will just be typeset. See my edit
    – Joseph Wright
    Jun 6, 2011 at 14:49
  • 1
    So why does \detokenize{\something}\bye comes out as '"something' in the typeset output? I'd expect '\something'.
    – AlexG
    May 30, 2013 at 8:07
  • 1
    @AlexG Font issue: the OT1 (Knuth original) font encoding doesn't have a \ in the appropriate slot, it has ". Assuming LaTeX, using \usepackage[T1]{fontenc} will solve this as in T1 encoding the slot is correct.
    – Joseph Wright
    May 30, 2013 at 9:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.