Take the 2-minute tour ×
TeX - LaTeX Stack Exchange is a question and answer site for users of TeX, LaTeX, ConTeXt, and related typesetting systems. It's 100% free, no registration required.

How does \detokenize work?

The e-TeX Short Reference Manual states:

When followed by a <general text>, expands to yield a sequence of character tokens of \catcode 10 (space) or 12 (other) corresponding to a decomposition of the tokens of the <balanced text> of the unexpanded <general text>; c.f. \showtokens. The effect is rather as if \scantokens applied to the <general text> within a régime in which only \catcodes 10 and 12 existed. Note that in order to preserve the boundaries between control words and any following letter, a space is yielded after each control word including the last.

What does that mean in plain English? I want to understand this snippet of code:

\def\task#1{\@task#1::\@nil}
\def\@task#1:#2:#3\@nil{%
  \if\relax\detokenize{#2}\relax
  ...

I think that \task is defined as a macro taking one argument which calls \@task. I don't know what \@nil means -- it is probably used to end the macro call. I also don't know what \relax and \detokenize do, but detokenize takes the second part of the argument for the task macro (the parts are separated with semicolons).

share|improve this question
add comment

2 Answers

up vote 18 down vote accepted

\task takes one argument, passing it to \@task which is defined in such a way that its arguments are delimited; if the call is

\@task xyz:AB:cde:u\@nil

the first argument is xyz, the second is AB and the third is cde:u. Here \@nil doesn't mean anything, it's just required by the syntax of \@task and TeX throws it away.

\relax is a primitive of TeX, its function is "do nothing". The test

\if\relax\detokenize{#2}\relax

is a safe way to determine if the argument #2 is empty. If it is, \detokenize{#2} expands to nothing, so \if compares the tokens \relax and \relax, which are indeed equal, so the "true" branch is followed, which starts immediately after the second \relax, up to and excluding \else. If #2 is not empty, say it's 30, \if will compare \relax with 3 which are different, so the "false" branch is followed, which starts after \else up to and excluding \fi.[1]

The similar construct \if\relax#2\relax does not work in all cases, because "all control sequences are equal as far as \if is concerned" [2]. It would not work if #2 was \relax (or any other control sequence, possibly followed by other tokens)! So we use \detokenize that, as explained also by Joseph, splits everything into a string. So, even in the weird case that #2 is \relax, \if would compare the token \relax with the character \, which are different.


[1] This is not strictly true, but it's an approximation of the truth sufficient for the purpose of this description.

[2] TeX by Topic (section 13.2.1)

share|improve this answer
    
Thank you again! I think that I understand it now. There are two more points that are not that clear to me: 1. Why are arguments for \@task split like that? xyz+AB+cde or xyz+AB:cde+u or xyz+AB+cde:u all seem to satisfy <part>:<part>:<part>. 2. Will \detokenize split \relax into \ r e l a x? –  ipavlic Jun 6 '11 at 14:50
1  
@ipavlic 1. TeX takes always the shortest token list that satisfies the requests (as long as the braces are balanced) so the first argument is anything up to the first colon, the second is anything from the first to the second colon and the third is anything from the second colon to \@nil. 2. Yes. –  egreg Jun 6 '11 at 14:57
    
Doesn't then the third argument has to be cde:u instead of u as written in the answer? This should not be nitpicking, I am really not sure enough to edit it. If I am wrong, please explain why. –  Patrick Häcker Oct 9 '12 at 17:19
    
@MMM You're right. –  egreg Oct 9 '12 at 17:28
add comment

The \detokenize primitive converts all of its input into category code 12 ('other') tokens, except for spaces which stay as category code 10 ('space'). It is used to convert tokens into strings. For example, if we have

\detokenize{\foo}

then \foo would be a control word, i.e. one token, which is converted into 5 string characters (\, f, o, o, ), all which are 'other' characters other than the space, which is a 'space'. Note the additional space at the end here: \detokenize adds a space after each 'control word' (see below).

I'll take your example one step at a time. First, \@nil is being used as a delimiter here. The definition of \@task means that the argument must contain :, then : again and finally \@nil. This is used to 'tidy up' awkward input

\task{stuff:more stuff}

will be turned into

 \@task stuff:more stuff::\@nil

which means #1 is stuff, #2 is more stuff and #3 is :. Thus it ensures that everything is 'used up': this is a common requirement.


To see \detokenize in action, try something like

\documentclass{article}
\usepackage[T1]{fontenc}
\edef\test{\detokenize{\test}}
\show\test
\begin{document}
\detokenize{\test}
\end{document}

A bit more on the addition of spaces by \detokenize, and indeed more generally. Whenever TeX writes something that can be one or more tokens as a 'string', it always inserts a space after each 'control word' (escape character followed by one or more 'letters') to avoid confusion. For example, with

\catcode`\X=12 % 'Other'
\def\foo{\barX}
\meaning\foo

you get

macro:-> \bar X

where the space is not there in the original but tells us that \bar is a control sequence followed by X, rather than printing just \barX, which looks like it's a different thing. This makes sense as TeX skips spaces after control words, so the two versions are equivalent as far as it is concerned.

This 'insert spaces' behaviour is respected by \detokenize, meaning that you get a space after each control word. The only place TeX doesn't insert these spaces is with \string, which can only be used on a single token and so in places where there is no possible 'confusion'.

share|improve this answer
    
Let me see if I understand it correctly. #1 from task expands to stuff:more stuff in the command \@task stuff:more stuff::\@nil. The purpose of the added :: is to enable input without semicolons, e.g. \task{1}. Now the semicolons will be concatenated to 1 to form 1:: which will satisfy the two-semicolons requirement of \@task. –  ipavlic Jun 6 '11 at 14:43
    
@ipavlic: #1 is never expanded, rather it is replaced by the input, but the result is as you say \@task stuff:more stuff::\@nil is executed. You are also right about the :: part - without this, you'd be in trouble with non-conforming input. –  Joseph Wright Jun 6 '11 at 14:47
1  
@ipavlic: \detokenize leaves the result in the input stream. Thus you end up with \detokenize{foo} => foo wherever it was used. In the current case, this is as part of a test, which will throw away any excess tokens. On the other hand, if you use it in a typesetting context then the material will just be typeset. See my edit –  Joseph Wright Jun 6 '11 at 14:49
    
@Joseph: I guess that the word expand has a special meaning in LaTeX, so "replace" would be more fitting. If I understand correctly, depending on the context, the commands behave a bit differently -- sometimes the parts of the content are replaced, and sometimes they are typeset "as they are". I think I am beginning to understand the concept of how LaTeX works. Thank you! –  ipavlic Jun 6 '11 at 14:57
    
@ipavlic: This is all TeX, not LaTeX per se. Expansion has a very precise meaning in TeX programming, and for low-level TeX questions like this one I think it's important to be careful about the terminology. –  Joseph Wright Jun 6 '11 at 15:02
show 9 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.