# \jobname, character codes and \detokenize

I recently wanted to compare \jobname to a string (with the xstring package) and was rather surprised that the string never matched. Tho internet told me that I had to use \detokenize on the string. \detokenize is an e-TeX command, that

when followed by a <general text>, expands to yield a sequence of character tokens of \catcode 10 (space) or 12 (other) corresponding to a decomposition of the tokens of the <balanced text> of the unexpanded <general text>;

Unfortunately, I do not own a copy of the TeXbook, so here are three questions:

• What is “balanced text”?
• Why does \jobname expand to the job name with “wrong” catcode?
• How did people compare \jobname to a string before e-TeX?
-
In the old days articles like these were useful tug.org/TUGboat/Articles/tb22-4/tb72wilson.pdf – Yiannis Lazarides Dec 9 '10 at 21:48
@Yiannis. In older versions of expl3 there was an expandable string-like test, based on the idea of dividing up the two arguments and using the \if primitive. It works quite nicely provided spaces are not important, but as we now require \pdfstrcmp it's been dropped. If it's of interest I can search back through the SVN and find it, but expect some mind-bending complexity! – Joseph Wright Dec 9 '10 at 21:54

## 1 Answer

Three questions there, but I think you'll be let off!

'Balanced text' means that the argument has to have balanced grouping characters, usually { and } pairs. This is because \detokenize requires an argument starting with a token with category code 1 (begin-group), in the same way as a token register. Indeed, you can do very similar things with a token register and with \detokenize:

\newtoks\mytoks
\def\test{stuff}
\mytoks\expandafter{\test}% \mytoks holds 'stuff' as letters
\detokenize\expandafter{\test}% Ouputs 'stuff' as 'other' tokens


On the category codes in \jobname, there are a number of places where you get a 'string' from TeX where everything except spaces has category code 12. You see the same with \the\<somedimen> and \meaning (more on the later in a moment). You'd have to ask DEK for the full story, but my understanding is that this 'string' approach is used so that no tokens are accidentally added to a control sequence name. There are places where if they were 'letters' then trouble might arise.

Finally, on the approach before e-TeX. As I said, \jobname is not the only place where you see 'string' output. In particular, \meaning does the same. So if you do

\def\testa{<whatever>}
\edef\testa{\meaning\testa}
\edef\testb{\jobname}
\edef\testb{\meaning\testb}
\ifx\testa\testb
...


the test will be true if the two names agree as lists of characters. There are variations on this method, see for example LaTeX's \strip@prefix, which can be used to make a 'string' without any prefix:

\makeatletter
\def\testa{<whatever>}
\edef\testa{\expandafter\strip@prefix\meaning\testa}% Now a 'string'


(As pointed out by Martin Scharrer, LaTeX's \@onelevel@sanitize is the same as the above: \@onelevel@sanitize\testa would be equivalent to the last line above. To show what is going on it's clearer to see the \meaning but in use you'd pick \@onelevel@sanitize.)

-
Thank you for the explanation. (btw, do you mean \strip@prefix in the last paragraph?) – Caramdir Dec 9 '10 at 21:41
@Caramdir: Oops, yes I did. Sorted. – Joseph Wright Dec 9 '10 at 21:51
Joseph and @Caramdir: Just to point that out: The last code can be simplified a little using \@onelevel@sanitize\testa. It is defined as macro:#1->\edef #1{\expandafter \strip@prefix \meaning #1}, i.e. identical to the posted code. – Martin Scharrer Mar 2 '11 at 18:13