TeX - LaTeX Stack Exchange is a question and answer site for users of TeX, LaTeX, ConTeXt, and related typesetting systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to find a way to use the e-TeX \scantokens command inside [the equivalent of] an \edef (well, actually an \xdef, but presumably this does not make any real difference since that's just the \global version of \edef). The reason I am doing this is that it seems to be needed to answer this question about hyperref and pdf outlines.

The problem seems to be that \scantokens doesn't actually expand to the tokens scanned; it actually expands to nothing, but arranges for the next tokens to be read from a pseudo-file named " ", which is then treated almost exactly like a real file, including the check for "runaway" conditions. (See page 10 of the e-TeX manual.)

There is one problem with this explanation, though: why would \input work, as it evidently does in the other question?

Anyway, here's a fairly minimal example:


% make \scantokens pseudo-files show up with name " " in the output


Which produces the output

Running `TeX' on `scantokens-in-edef-minimal' with ``pdftex  -interaction=nonstopmode "\input" "scantokens-in-edef-minimal"''
This is pdfTeX, Version 3.1415926-1.40.11 (MiKTeX 2.9)
entering extended mode
(e:\home\tex\scantokens-in-edef-minimal.tex ( )
Runaway definition?
! File ended while scanning definition of \foo.
<inserted text> 
l.6 \edef\foo{\scantokens{Test}
! Too many }'s.
l.6 \edef\foo{\scantokens{Test}}

! Emergency stop.
<*> \input scantokens-in-edef-minimal

!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on scantokens-in-edef-minimal.log.

TeX exited abnormally with code 1 at Thu Jan 27 12:24:57

Is it perhaps possible to somehow do the expansion with macros and then do a \def instead of the \edef, so the expansion would be finished before the definition began, and the "runaway definition" condition avoided?

share|improve this question
up vote 14 down vote accepted

In short, yes. Try




in which the \noexpand ‘hides’ the EOF marker by turning it into \relax.

You might like to take a look at the definition of expl3's \tl_rescan:nn and \tl_set_rescan:Nnn, both wrappers for \scantokens based on the ideas in Heiko Oberdiek's catchfile package.

Then you can write

\tl_set_rescan:Nnn \foo { «catcode setup» } { «rescanned tokens» }


\tl_rescan:nn { «catcode setup» } { «tokens to re-scan (in a group)» }

The latter cannot be used inside an \edef but can otherwise come in handy when you're processing a macro argument that has already been tokenised. In practise I find the set_rescan version a little more convenient.

share|improve this answer
And if you were using luatex, you could use \scantextokens instead of the \everyeof hack. – Taco Hoekwater Jan 28 '11 at 9:06
@Taco—Indeed! Sorry I omitted it :) – Will Robertson Jan 28 '11 at 10:42
Cool; I'll relay this answer back to the asker of the other question. I was afraid it would be necessary to essentially implement \edef/\xdef from scratch! – SamB Jan 28 '11 at 18:05
Now if only I could understand how making the } act like \relax would fool e-TeX into finishing the \scantokens without doing the runaway checks, then I'd ... well, I guess I might just have a headache, actually :-). – SamB Jan 28 '11 at 19:30
@SamB—Perhaps this thread may be instructive: groups.google.com/group/comp.text.tex/browse_thread/thread/… – Will Robertson Jan 30 '11 at 2:17

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.