Does `\noexpand` have to be a primitive?

Question

Background: I'm trying to deepen my understanding of TeX and how the intricacies work (spurred a little by this place, I should say, and hoping one day to not be a "Cargo Cult Programmer" - see When to use \edef, \noexpand, and \expandafter? both for my other attempts to deepen my knowledge and the - justified - accusation of being a CCP). My source is TeX-by-Topic, but I'm not good at reading from first to last and dip in and out so don't always understand things fully when I read them.

I feel I'm beginning to get the idea of expansion, but still a few things elude me. To try to make an answerable question out of this, let me focus on \noexpand. According to TeX-by-Topic, this is expandable and sets the next token to temporarily be \relax. It's the "temporarily" that I'm finding hard to get my head around. If one wanted to emulate \noexpand as a macro, then the best I could think of was:

\def\noexpand#1{\let\tempa#1%
\let#1\relax
#1%
\let#1\tempa}

but I suspect that this would cause problems in an "expand-until-no-expandables-left" context (such as following an \if). So, to make this a definite question: Can \noexpand be implemented as a macro, or is there some black magic that requires it to be a primitive?

(The picture that I have in mind when I think of \noexpand is that TeX goes in to a ravenous eating mode wherein it chomps up a load of input in its mouth and grinds it up as much as possible before swallowing it, but \noexpand wraps its stuff in an unchewable coating. However, that coating is not immune to the stomach acids. Unfortunately, sometimes TeX (not being well brought up) does a little belch and some of what it ate comes back up to be reconsumed. At this point, the stuff previously protected by \noexpand has had its coating removed by the acids and is now chewable.)

Answer 1

(Further updated answer, with a look deep down the Lion's Mouth in the last section.)

Joseph gave an answer (“No!”) to your actual question; I'll try to get your (and my) head around the “temporarily”. Let's start from TeX-by-Topic, bottom of page 129:

The \noexpand command is expandable, and its expansion is the following token. The meaning of that token is made temporarily equal to \relax, so that it cannot be expanded further.

At first I dismissed this as a colourful description, but one can easily see that TeX-by-Topic is right: The following short plain TeX file

\def\foo{bar}
\expandafter\show\noexpand\foo
\bye

prints

> \foo=\relax.
<recently read> \notexpanded: \foo 

to the terminal, so \foo is indeed “made temporarily equal to \relax”. What I find very interesting is the \notexpanded: \foo; this \notexpanded isn't mentioned in the TeXbook or in TeX-by-Topic, and I only found one occurrence in Knuth's torture test for TeX. (Another test is to put \tracingcommands=2 \noexpand\foo into your TeX file; then you'll find a \relax in your log file.)

In the TeXbook there's a description that I find easier chewable than the one from TeX-by-Topic (since it doesn't contain “temporarily”): On page 214 it says

\noexpand⟨token⟩. The expansion is the token itself; but that token is interpreted as if its meaning were ‘\relax’ if it is a control sequence that would ordinarily be expanded by TeX’s expansion rules.

So what does “temporarily” mean?

In practise, “temporarily equal to \relax” means the following: When TeX expands \noexpand\foo, the result is \foo with the temporary meaning \relax, and as soon as TeX continues expanding, \foo gets back its original meaning. A typical use case of \noexpand, from Tex-by-Topic (page 130):

\edef\one{\def\noexpand\two{\the\prevdepth}}

Inside the \edef, TeX doesn't expand \two but continues with the next token { (which is also not expandable). The \noexpand is needed since otherwise TeX would try to expand \two. This would cause an error if \two is undefined, and it would cause much more trouble if \two has been defined before. (Before \def one doesn't need a \noexpand since \def is not expandable.) If you now \show the \one, then you see that the real \two, not a \relaxed one is inside \one:

> \one=macro:
->\def \two {-1000.0pt}.

What's happening behind the scenes?

Let me first say this in the picture of TeX's mouth and stomach. In general, TeX continues chewing on a token list in its mouth as long as the first token is expandable. When the first token is unexpandable, it's swallowed for further processing in the stomach. Now \noexpand indeed wraps the token following it in some protective coating. However, this coating is chewable; figure that it's made of sugar. Thus, when TeX chews on such a coated token, the coating is removed (and the token is expandable again), but TeX gets a sensory flash and thinks “Wow, this tastes like \relax, I want to swallow it.” And sure enough, if the now uncoated token is the first in TeX's mouth (think of “first” as “closest to the throat”), then it is swallowed.

Two examples (the 2nd one being rather academic):

1. You have, e.g.,

   \edef\bar{\noexpand\foo\anothercs}
   

   Then inside the \edef, \noexpand\foo expands to a \relaxed \foo. This is converted back to \foo, immediately swallowed, and TeX continues with expanding the next token \anothercs.

2. You expand \noexpand\foo twice, e.g. with

   \def\foo{bar}
   \expandafter\expandafter\expandafter\show\noexpand\foo
   \bye
   

   Then on the terminal you get

   > \foo=macro:
   ->bar.
   

   Thus, expanding \noexpand\foo once gives a \relaxed \foo (as seen above), and the expansion of the \relaxed \foo is again the original \foo. And this is not swallowed since it's not the first token!

Disclaimer: The above I found out by observation; it might be that some details are not entirely correct.

Some final remarks

Your attempt at defining \noexpand as macro fails for several reasons. One technical point is that \let#1\relax is not a good idea if #1 isn't a control sequence. Another point is that it's not really helpful to do \let#1\relax #1; this has the same effect as \relax as far as I can tell. (And as pointed out in the other answers, \let is not expandable.)

Answer 2

Your macro-based approach will fail as you use assignments (\let): these cannot be used in an expansion context. While TeX is Turing-compete, this says nothing about expansion. It's not possible to alter the meaning of something within an expansion (at least without LuaTeX). So the answer to the question is 'Yes, this has to be a primitive'.

Answer 3

I'm pretty sure that \noexpand must be a primitive. It causes the following token to behave like \relax in that \relax is not expandable.

\edef\foo{\relax}

has the same effect as

\def\foo{\relax}

\noexpand lets you do that with any token.

\edef\bar{\noexpand\foo}

causes the replacement text of \bar to be \foo. Without the \noexpand it would be \relax.

As Joseph's answer (which appeared as I wrote this) points out, your version of \noexpand is not expandable because it uses assignments.

Consider how it behaves in the \edef below.

\def\foo{ABC}
\edef\bar{\noexpand\foo}

First, \noexpand\foo would expand to

\let\tempa\foo \let\foo\relax \foo \let\foo\tempa \foo

now since we are expanding, \foo would expand everywhere to

\let\tempa ABC \let ABC\relax ABC \let ABC\tempa ABC