Does \noexpand have to be a primitive?

Question

Background: I'm trying to deepen my understanding of TeX and how the intricacies work (spurred a little by this place, I should say, and hoping one day to not be a "Cargo Cult Programmer" - see When to use \edef, \noexpand, and \expandafter? both for my other attempts to deepen my knowledge and the - justified - accusation of being a CCP). My source is TeX-by-Topic, but I'm not good at reading from first to last and dip in and out so don't always understand things fully when I read them.

I feel I'm beginning to get the idea of expansion, but still a few things elude me. To try to make an answerable question out of this, let me focus on \noexpand. According to TeX-by-Topic, this is expandable and sets the next token to temporarily be \relax. It's the "temporarily" that I'm finding hard to get my head around. If one wanted to emulate \noexpand as a macro, then the best I could think of was:

\def\noexpand#1{\let\tempa#1%
\let#1\relax
#1%
\let#1\tempa}

but I suspect that this would cause problems in an "expand-until-no-expandables-left" context (such as following an \if). So, to make this a definite question: Can \noexpand be implemented as a macro, or is there some black magic that requires it to be a primitive?

(The picture that I have in mind when I think of \noexpand is that TeX goes in to a ravenous eating mode wherein it chomps up a load of input in its mouth and grinds it up as much as possible before swallowing it, but \noexpand wraps its stuff in an unchewable coating. However, that coating is not immune to the stomach acids. Unfortunately, sometimes TeX (not being well brought up) does a little belch and some of what it ate comes back up to be reconsumed. At this point, the stuff previously protected by \noexpand has had its coating removed by the acids and is now chewable.)

Answer 1

(Further updated answer, with a look deep down the Lion's Mouth in the last section.)

Joseph gave an answer (“No!”) to your actual question; I'll try to get your (and my) head around the “temporarily”. Let's start from TeX-by-Topic, bottom of page 129:

The \noexpand command is expandable, and its expansion is the following token. The meaning of that token is made temporarily equal to \relax, so that it cannot be expanded further.

At first I dismissed this as a colourful description, but one can easily see that TeX-by-Topic is right: The following short plain TeX file

\def\foo{bar}
\expandafter\show\noexpand\foo
\bye

prints

> \foo=\relax.
<recently read> \notexpanded: \foo 

to the terminal, so \foo is indeed “made temporarily equal to \relax”. What I find very interesting is the \notexpanded: \foo; this \notexpanded isn't mentioned in the TeXbook or in TeX-by-Topic, and I only found one occurrence in Knuth's torture test for TeX. (Another test is to put \tracingcommands=2 \noexpand\foo into your TeX file; then you'll find a \relax in your log file.)

In the TeXbook there's a description that I find easier chewable than the one from TeX-by-Topic (since it doesn't contain “temporarily”): On page 214 it says

\noexpand⟨token⟩. The expansion is the token itself; but that token is interpreted as if its meaning were ‘\relax’ if it is a control sequence that would ordinarily be expanded by TeX’s expansion rules.

So what does “temporarily” mean?

In practise, “temporarily equal to \relax” means the following: When TeX expands \noexpand\foo, the result is \foo with the temporary meaning \relax, and as soon as TeX continues expanding, \foo gets back its original meaning. A typical use case of \noexpand, from Tex-by-Topic (page 130):

\edef\one{\def\noexpand\two{\the\prevdepth}}

Inside the \edef, TeX doesn't expand \two but continues with the next token { (which is also not expandable). The \noexpand is needed since otherwise TeX would try to expand \two. This would cause an error if \two is undefined, and it would cause much more trouble if \two has been defined before. (Before \def one doesn't need a \noexpand since \def is not expandable.) If you now \show the \one, then you see that the real \two, not a \relaxed one is inside \one:

> \one=macro:
->\def \two {-1000.0pt}.

What's happening behind the scenes?

Let me first say this in the picture of TeX's mouth and stomach. In general, TeX continues chewing on a token list in its mouth as long as the first token is expandable. When the first token is unexpandable, it's swallowed for further processing in the stomach. Now \noexpand indeed wraps the token following it in some protective coating. However, this coating is chewable; figure that it's made of sugar. Thus, when TeX chews on such a coated token, the coating is removed (and the token is expandable again), but TeX gets a sensory flash and thinks “Wow, this tastes like \relax, I want to swallow it.” And sure enough, if the now uncoated token is the first in TeX's mouth (think of “first” as “closest to the throat”), then it is swallowed.

Two examples (the 2nd one being rather academic):

1. You have, e.g.,

   \edef\bar{\noexpand\foo\anothercs}
   

   Then inside the \edef, \noexpand\foo expands to a \relaxed \foo. This is converted back to \foo, immediately swallowed, and TeX continues with expanding the next token \anothercs.

2. You expand \noexpand\foo twice, e.g. with

   \def\foo{bar}
   \expandafter\expandafter\expandafter\show\noexpand\foo
   \bye
   

   Then on the terminal you get

   > \foo=macro:
   ->bar.
   

   Thus, expanding \noexpand\foo once gives a \relaxed \foo (as seen above), and the expansion of the \relaxed \foo is again the original \foo. And this is not swallowed since it's not the first token!

Disclaimer: The above I found out by observation; it might be that some details are not entirely correct.

Some final remarks

Your attempt at defining \noexpand as macro fails for several reasons. One technical point is that \let#1\relax is not a good idea if #1 isn't a control sequence. Another point is that it's not really helpful to do \let#1\relax #1; this has the same effect as \relax as far as I can tell. (And as pointed out in the other answers, \let is not expandable.)

Answer 2

Your macro-based approach will fail as you use assignments (\let): these cannot be used in an expansion context. While TeX is Turing-compete, this says nothing about expansion. It's not possible to alter the meaning of something within an expansion (at least without LuaTeX). So the answer to the question is 'Yes, this has to be a primitive'.

Answer 3

The following answer consists of two parts.

1. \noexpand's IMPLEMENTATION. In this part I'll outline how a TeX engine might implement the \noexpand primitive. I'll also establish some terminology that will be used in part 2. Part 1 answers the question posed by OP of how to interpret the following statement

   \noexpand sets the next token to temporarily be \relax.

   The implementation I will present is speculative. In other words, I'm not claiming that this is how any of the existing TeX engines are implemented. All I'm saying is that it would be possible to implement a TeX engine along the lines described in this part, and that the resulting implementation will behave in a way that agrees with the existing implementations, as far as \noexpand is concerned.

2. STEP-THROUGH EXAMPLES. In this part I'll apply the theory of the first part to four snippets of TeX code, each of which demonstrates how \noexpand interacts with some other TeX primitive. The four other primitives are, in order of presentation: \edef, \expandafter, \ifx, and e-TeX's \numexpr.

   For every one of these primitives, I will

   (a) Suggest how it might be implemented by a TeX engine.
   (b) List a TeX code snippet that exhibits an interaction between this primitive and \noexpand.
   (c) Step through the execution of this snippet, given the \noexpand implementation described in part 1 and the suggested implementation of the other primitive.

---

1. \noexpand'S IMPLEMENTATION

Every token t has a value. The value of t is that which can be \let to some control-sequence or active character, as in \let\u=t. We shall refer to the values initially assigned to TeX's primitive control-sequence \<cs name> by <cs name>; for instance, the value initially assigned to \meaning is meaning, and the value initially assigned to \relax is relax, etc. These will be referred to as the primitives' default values.

The TeX engine maintains a table mapping token names (i.e. what \string t expands to) to an ordered pair (v,x), where v is the token's value, and x is a boolean flag to be described below. We shall refer to x as the token's "temporarily non-expandable" flag. When an entry is added to the table, x is automatically set to false, but this can be changed later, as explained below.

This table can be accessed only via an API. This API consists of three functions that have the following signatures:

void temp_nonexpand(string tk_name)
value_obj get_value(string tk_name)
int compare_temp_nonexpand(string tk_name1, string tk_name2)

• temp_nonexpand takes a token name as argument, and sets the "temporarily non-expandable" flag associated with this name to true. There is no way to explicitly reset this flag to false; this is done automatically by get_value and compare_temp_nonexpand, as explained below.
• get_value takes a token name as argument, and returns (a pointer to) the value mapped to this name. When get_value(...) is called and passed the name of a control-sequence whose "temporarily non-expandable" flag is true, get_value returns the value relax, regardless of what value is actually mapped to this control-sequence, and resets the token's "temporarily non-expandable" flag to false. Thus, the next time get_value will be called with the same name, the value actually mapped to this name will be returned. This is the meaning of the quote cited in the beginning of this answer.
• compare_temp_nonexpand takes two token names as arguments, and returns 1 if both tokens' "temporarily non-expandable" flags are true, -1 if both tokens' "temporarily non-expandable" flags are false, and 0 otherwise (i.e. one of the flags is true, and the other is false, not necessarily in this order). Like get_value, compare_temp_nonexpand resets both tokens' "temporarily non-expandable" flag to false. This function is used solely by the implementation of ifx; see section 2 below for details.

Every value is either expandable or non-expandable. The user has no control over this property; it is decreed in the TeXbook (20th printing, Addison-Wesley 1991) as follows: the expandable values are the macros, and the default values of some of TeX's primitive control-sequences. A complete list of the expandable primitives can be found on pp. 212-215 of the TeXbook, but here are some examples.

• The expandable values include the default values of the following primitives: \expandafter, \ifx, \meaning, \noexpand, \string, and \the.
• The non-expandable values include the default values of the following primitives: \def, \edef, \relax, and \show.

When a token's value is expandable, we say that the token itself is expandable. To expand the token is to expand its value. What "expanding a value" entails depends on the value. For example, what it means to expand noexpand is this. Denote by u the next token on the input stream. If uis expandable, u's "temporarily non-expandable" flag is set to true by executing temp_nonexpand(\string u).

Some of the non-expandable primitives are executable commands (e.g. all the primitives listed in the last bullet-point), and some are not (e.g. registers, parameters). There is no real difference between "executing" and "expanding", except we apply these verbs to different objects: the former to non-expandable command primitives, whereas the latter to macros and expandable primitives. We shall also apply the word "execute" to internal functions such as get_value, as we did in the end of the last paragraph.

---

2. STEP-THROUGH EXAMPLES

Now let's put all this theory to practice, by stepping through four examples. Each example shows how \noexpand interacts with some other TeX primitive, namely

1. \edef
2. \expandafter
3. \ifx
4. \numexpr (an e-TeX primitive)

All the examples assume that the all the primitives featured in them have their default values.

Example 1: \edef

We shall only be concerned with how edef constructs the internal representation of the defined macro's replacement text from the tokens specified inside the curly braces in the input stream. edef implements the following algorithm to construct the replacement text.

1. Initialize the token list replacement_text to the empty list.
2. Remove the first token from the input stream, and assign it to the variable t.
3. Repeat as long as t is not the closing brace:
   1. Get t's value: v := get_value(\string t).
   2. If v is non-expandable, append the token to the replacement text: replacement_text.append(t).
   3. If v is expandable, expand v.
   4. Remove the first token from the input stream, and assign it to t.

Now consider the following TeX code (taken from Hendrik Vogt's answer):

\edef\bar{\noexpand\foo\anothercs}

Let's step through the construction of the replacement text.

1. The replacement_text token-list is initialized to the empty list.
2. \noexpand is removed from the input stream and assigned to t.
3. \noexpand's value is assigned to v: v := get_value("noexpand"), so now v ==nonexpand.
4. Since noexpand is expandable, it is expanded, by executing temp_nonexpand("foo"). This turns on \foo's "temporarily non-expandable" flag.
5. \foo is removed from the input stream, and assigned to t.
6. We set v := get_value("foo"). Since \foo's "temporarily non-expandable" flag is true when get_value is invoked, v==relax, and the flag is turned off.
7. Since relax is non-expandable, \foo is appended to the replacement_text list.
8. \anothercs is removed from the input stream, and assigned to t.
9. If \anothercs's value is non-expandable, \anothercs will be appended to the replacement_text list; otherwise, it will be expanded, and so on.

Example 2: \expandafter

expandafter is implemented as follows. It is assumed that there is a dedicated global token stack s intended for expandafter's use. s can be assumed to be initially empty, though this fact will not be used here.

1. Remove the first token from the input stream, and push it on the stack s.
2. Remove the first token from the input stream, and assign it to the variable t.
3. Retrieve t's value: v := get_value(t).
4. If v is non-expandable, push t back to the front of the input stream. Otherwise, expand it.
5. Pop the token at the top of s, and add it to the front of the input stream.

Now consider the following TeX manuscript (taken from Hendrik Vogt's answer):

\def\foo{bar}%
\expandafter\expandafter\expandafter\show\noexpand\foo%
\bye

Let's step through the expansion of the second line.

1. \expandafter is removed from the input stream, and its value, expandafter, is retrieved. Since expandafter is expandable, it is expanded, as follows.

   1. The 2nd occurrence of \expandafter is removed from the input stream, and is pushed on s, so the stack now holds only this one token.
   2. The 3rd occurrence of \expandafter is removed from the input stream, and assigned to t.
   3. We set v := get_value("expandafter"). Now v ==expandafter.

   4. Since expandafter is expandable, it is expanded, as follows.
      1) \show is removed from the input stream, and pushed on s, so that s now holds \show\expandafter, where the top of the stack is on the left.
      2) \noexpand is removed from the input stream, and assigned to t.
      3) We set v := get_value("noexpand"). Now v ==noexpand.
      4) Since nonexpand is expandable, it is expanded, by executing temp_nonexpand("foo"). This turns on \foo's "temporarily non-expandable" flag.
      5) s is popped, and the popped item, namely \show, is added to the front of the input stream, which now looks as follows:

      \show\foo
      

   5. s is popped, and the popped item, namely \expandafter, is added to the front of the input stream, which now looks as follows:

      \expandafter\show\foo
      

2. \expandafter is removed from the input stream, and its value, expandafter, is retrieved. Since expandafter is expandable, it is expanded, as follows.

   1. \show is removed from the input stream, and is pushed on s.
   2. \foo is removed from the input stream, and assigned to t.
   3. We set v := get_value("foo"). Since at the time get_value is called, \foo's "temporarily non-expandable" flag has been true since step 1.4.4, v ==relax, and \foo's "temporarily non-expandable" flag is turned off.
   4. Since relax is non-expandable, \foo is pushed back to the front of the input stream.

   5. s is popped, and the popped item, namely \show, is added to the beginning of the input stream. The input stream is now:

      \show\foo
      

3. \show is removed from the input stream, and its value, show, is retrieved. Since show is a command, it is executed: \foo's value is retrieved, as follows. First we set v := get_value("foo"), and then a representation of v is written to the log file. Since the get_value call in step 2.3 has turned off \foo's "temporarily non-expandable" flag, v == (macro:->bar).

Example 3: \ifx

ifx is implemented as follows.

1. Remove the first two tokens from the input stream, and assign them to the variables t and u, respectively.
2. Set c := compare_temp_nonexpand(\string t, \string u). As a side-effect, this operation sets the "temporarily non-expandable" flags of both tokens to false.
3. If c == ...
   • 1, i.e. if both tokens were temporarily non-expandable, the test evaluates to true.
   • 0, i.e. if exactly one of the tokens was temporarily non-expandable, the test evaluates to false.
   • -1, i.e. if neither token was temporarily non-expandable, the test evaluates according to the usual rules (see the TeXbook, p. 210).
4. Once the test's value is known, ifx's expansion proceeds as usual (see the TeXbook, p. 213).

For example, the following TeX manuscript:

\def\foo{foo}%
\def\bar{bar}%
\expandafter\expandafter%
    \expandafter\ifx%
        \expandafter\noexpand\expandafter\foo\noexpand\bar yes%
    \else no%
    \fi%
\bye

typesets "yes", even though \foo and \bar don't have the same \meaning, because at the time the test is performed, both tokens are temporarily non-expandable.

However, the following manuscript:

\def\foo{foo}%
\expandafter\ifx\noexpand\foo\relax yes\else no\fi%
\bye

typesets "no", even though \noexpand\foo and \relax have the same meaning (try \expandafter\meaning\noexpand\foo), because at the time the test is performed, \foo's "temporarily non-expandable" flag is true, whereas \relax's "temporarily non-expandable" flag is false.

The following caveat should be noted. ifx is the only TeX primitive whose implementation invokes compare_temp_nonexpand. In particular, neither \if, nor any of the other conditionals (e.g. \ifcat) do so. Therefore, if you substitute \if for \ifx above, both tests will evaluate to true, since the temporarily non-expandable tokens will be interpreted as \relax.

Example 4: \numexpr

numexpr is a non-expandable primitive command that is not a part of core TeX, but of the e-TeX extension, which is implemented by pdftex. I shall describe a simplified version of this algorithm, which behaves somewhat differently than the original, but is sufficiently close to it to exemplify how numexpr and noexpand interact for real.

The version of numexpr that will be described below takes as input a sequence of non-negative integers interspersed with binary arithmetic operators, e.g. 1+20*3, and assigns the result of the expression to a new integer register \return. No whitespace is allowed inside the expression. Any token that is neither a digit nor a binary arithmetic operator marks the end of numexpr's input. The operations are executed in the textual order, so, for instance, the expression above will evaluate to 63 (= (1 + 20) * 3) rather than to 61. numexpr removes its input from the input stream. If the expression is followed by a token whose value is relax, this token is also removed from the input stream (which is how the real numexpr primitive behaves; see p. 8 of the e-Tex manual).

The numexpr algorithm makes use of two auxiliary functions: get_next_nonexpandable and parse_nonneg_int.

get_next_nonexpandable

get_next_nonexpandable gets the next non-expandable token from the input stream by repeatedly expanding the first item, if necessary. The token's value is also returned (this point is critical if the example below is to behave the way it does in real life). Its signature is

(token, value) get_next_nonexpandable()

and its implementation is:

1. Remove the first token from the input stream, and assign it to the variable t.
2. Get t's value: v := get_value(\string t).
3. Repeat as long as v is expandable:
   1. Expand v.
   2. Remove the first token from the input stream, and assign it to the variable t.
   3. v := get_value(\string t)
4. Return (t, v).

parse_nonneg_int

parse_nonneg_int removes the longest possible sequence of digits from the beginning of the input stream, and returns the non-negative integer represented by this string of digits. Its signature is

int parse_nonneg_int()

and its implementation is:

1. Initialize the non-negative integer variable D to 0. This variable will accumulate the intermediate calculations, and will eventually hold the number to be returned.
2. Get the next non-expandable token from the beginning of the input stream: (t, v):= get_next_nonexpandable().
3. Repeat as long as v is of the form (character, catcode) and the character is a digit.
   1. Denote v's character by c.
   2. Convert c to an integer d.
   3. Set D := D*10 + d.
   4. Get the next non-expandable token from the beginning of the input stream: (t, v):= get_next_nonexpandable().
4. Push t back to the beginning of the input stream.
5. Return D.

numexpr

Using get_next_nonexpandable and parse_nonneg_int, the simplified version of numexpr's algorithm is:

1. Initialize the integer variable i := 0. This variable will accumulate the intermediate calculations, and will eventually hold the result of the entire arithmetic expression.
2. Initialize the character variable op:='+'. This variable will hold the most recently encountered arithmetic operator.
3. Get the next non-expandable token from the input stream: (t,v) := get_next_nonexpandable().
4. Repeat as long as v is of the form (character, catcode), and the character is either a digit or a binary arithmetic operator.
   1. Denote v's character by c.
   2. If c is a digit:
      1) Push t back to the beginning of the input stream.
      2) Parse the longest possible non-negative integer from the beginning of the input stream: j := parse_nonneg_int().
      3) Perform the operation designated by op on the arguments i and j.
      4) Set op to null.
   3. If c is a binary arithmetic operator:
      1) If op != null, report an error ("2 consecutive operators!"), and exit the TeX engine.
      2) If op == null, assign op := c.
   4. Get the next non-expandable token from the input stream: (t,v) := get_next_nonexpandable().
5. If op != null, report an error ("Dangling operator!"), and exit the TeX engine.
6. If v !=relax, push t back to the beginning of the input stream.
7. Allocate a new integer register: \newcount\result.
8. Assign i to the integer register: \result{<tokenized representation of i>}.
9. Add the token \result to the beginning of the input stream.

Now consider the following TeX code (which is a slightly simplified version of the code proposed by Martin Scharrer in a comment to Hendrik's answer):

\the\numexpr 1+1\noexpand\empty\relax

Let's step through the processing of this line, starting with \numexpr. In other words, the TeX engine is in the middle of expanding the, and has just begun executing numexpr, so that the input stream is currently 1+1\noexpand\empty\relax. We assume that \empty is a macro that is defined as follows: \def\empty{}.

1. Initialize the integer variable i:=0, and a character variable op:='+'.
2. (t, v) := get_next_nonexpandable(). t is now the token 1 with value ('1', 12) (12 is the default catcode for digits).
3. since 1 is a digit, t is pushed back to the front of the input stream, and we set j := parse_nonneg_int(). j is now equal to the integer 1, and the input stream is +1\noexpand\empty\relax.
4. Assign i := i + j. i is now equal to 1.
5. (t, v) := get_next_nonexpandable(). t is now the token + with value ('+',12) (12 is the default catcode for the character '+').
6. Since + is a binary arithmetic operator, we assign op:='+'.
7. (t, v) := get_next_nonexpandable(). t is now the token 1 with value ('1', 12).

8. since 1 is a digit, t is pushed back to the front of the input stream, and we set j := parse_nonneg_int(). Let's step through the execution of parse_nonneg_int.

   1. The nonnegative integer variable D is initialized to 0.
   2. (t, v) := get_next_nonexpandable(). t is now the token 1, and v is ('1', 12).
   3. Since 1 is a digit, it is converted to the corresponding integer d:=1, and we set D:=D*10+d. D is now equal to 1.
   4. (t, v) := get_next_nonexpandable(). Let's step through the execution of get_next_nonexpandable.
      1) The first token is removed from the input stream and assigned to the variable t. So now t == \noexpand.
      2) t's value is retrieved and assigned to the variable v: v:=get_value("noexpand"). So now v ==noexpand.
      3) Since noexpand is expandable, it is expanded, by executing set_temp_nonexpand("empty"). So now \empty is flagged as temporarily non-expandable.
      4) The first token of the input stream is removed from the input stream and assigned to t. So now t == \empty.
      5) t's value is retrieved and assigned to v: v:=get_value("empty"). Since at the time get_value is called, \empty is flagged as temporarily non-expandable, get_value returns relax, and turns off \empty's "temporarily non-expandable" flag. So now v ==relax, and \empty is no longer temporarily non-expandable.
      6) Since relax is non-expandable, (t, v), i.e. (\empty,relax), is returned.

   5. Since v==relax is not of the form (character, catcode), t is pushed back to the beginning of the input stream, and parse_nonneg_int returns D, i.e. the integer 1. The input stream is now

      \empty\relax
      

9. Now i == 1, j == 1, and op == '+', and we set i := i + j, so that now i == 2, and we set op := null.

10. (t, v) := get_next_nonexpandable(). Let's step through the execution of get_next_nonexpandable.
    1. The first token of the input stream is removed from the input stream, and assigned to t, so that t == \empty.
    2. t's value is retrieved and assigned to v: v := get_value("empty"). Recall that \empty's "temporarily non-expandable" flag has been false since the end of step 8.4.5. So get_value returns \empty's actual value, namely the empty macro.
    3. Since v is a macro, it is expandable, so \empty is expanded. However, since \empty's body is empty, the input stream remains intact (and, of course, so does the TeX engine's internal state). So after \empty's expansion the input stream consists of a single token: \relax.
    4. The first token is removed from the input stream, and assigned to t, so that t == \relax.
    5. t's value is retrieved and assigned to v: v := get_value("relax"). So now v ==relax.
    6. Since relax is non-expandable, get_next_nonexpandable returns (t, v), i.e. (\relax,relax).
11. Since v ==relax, numexpr has reached the end of its loop (step 4 in numexpr's algorithm). So we allocate a new integer register \result: \newcount\result.
12. We assign i to \result: \result{2}.

13. We add the token \result to the beginning of the input stream, so that now the input stream consists of a single token: \result, and this input stream is "handed over" to the the primitive to act on. So the end result will be as though the TeX engine was originally presented with the input stream

    \the\result
    

    with \result being an integer register holding the value 2.

14. The text "2" is typeset.

Answer 4

I'm pretty sure that \noexpand must be a primitive. It causes the following token to behave like \relax in that \relax is not expandable.

\edef\foo{\relax}

has the same effect as

\def\foo{\relax}

\noexpand lets you do that with any token.

\edef\bar{\noexpand\foo}

causes the replacement text of \bar to be \foo. Without the \noexpand it would be \relax.

As Joseph's answer (which appeared as I wrote this) points out, your version of \noexpand is not expandable because it uses assignments.

Consider how it behaves in the \edef below.

\def\foo{ABC}
\edef\bar{\noexpand\foo}

First, \noexpand\foo would expand to

\let\tempa\foo \let\foo\relax \foo \let\foo\tempa \foo

now since we are expanding, \foo would expand everywhere to

\let\tempa ABC \let ABC\relax ABC \let ABC\tempa ABC

Answer 5

Some thoughts in the context of LuaTeX. (to be fair, this is a bit "cheating" in that as you give Lua more and more power, more things become possible to implement)

Does the OP's implementation (with some modifications) work?

The original implementation can be changed to this... _{(assume this is \ExplSyntaxOn environment so no spurious space)}

\def\noexpand #1{
 \immediateassignment \let \tmpa #1 
 \immediateassignment \let #1 \relax
 #1
 \immediateassignment \let #1 \tmpa
}

It satisfies these 2 properties that the real \noexpand satisfies

1. in edef context, the token get collected into the \edef result
2. in normal execution context, it does nothing.

Problem (the obvious ones first)

• what if the next token is { or }?
• what if the next token is \let itself?
• (more generally speaking, if the next token is unexpandable, \noexpand should do nothing with it)

Assume these problems are fixed. What other problems are there?

Apart from those 2 properties:

3. the real \noexpand also takes only 1 expansion step to expand to the \notexpanded: token.

As a normal macro it would take at least 2.

What if I don't care about the exact number of expansion steps?

That would make you severely limited in what you can do with TeX. In particular, since \expandafter expands the following tokens exactly once.

Okay, what if that problem is fixed with luadef?

In LuaTeX, currently you also have the power to make the single-expansion of any token do whatever you want; and you can abuse token.scan_toks() a bit to do arbitrary expansion; combined with \immediateassignment and tex.runtoks you can execute (almost) arbitrary TeX code 1-step expandably.

So you make \noexpand \__a assigns \relax to \__a, then expands in 1 step to \__a \immediateassignment \let \__a \relax. Does that work?

Problem: the real \noexpand only expands to 1 token.

It's not too difficult to find out a case where this causes a problem. e.g. if you do a \futurelet, you'll get \immediateassignment instead of the real next token.

What if that problem is fixed?

I can see one way this problem can be fixed, namely making \noexpand \__a expands 1-step to a new token (with token.new()) that has string representation identical to \__a; but nevertheless is a different token, and when expanded/executed behaves accordingly.

Side note, "behaves accordingly" here in \noexpand's context also means

4. when it's once-expanded (e.g. with \expandafter, or "number scanners" such as \number/\ifnum etc. or dimen scanner etc.), it "appears" to be unexpandable but expands to the original token.
5. The other \__a tokens must not change. In other words, only that new token can have the \relax meaning, so you must not actually do \let \__a \relax.

• (when it's inside an \edef or in normal execution context, see above.)

Note that this requires the new token to be able to somehow distinguish whether or not it's inside an \edef. As far as I can tell even in current LuaTeX it's not possible without any side effect; although patching \edef etc. is an option.

Okay, assume all of the above. Does it work, then?

Problem: There's one last special property of \notexpanded: tokens:

6. when it's scanned as a command argument, it loses its \notexpanded: property.

For example

\def\showwrapper #1{\show #1}
\expandafter \show \noexpand \__a  % shows \notexpanded
\expandafter \showwrapper \noexpand \__a  % shows the meaning of the original token

Only by putting it untouched on the input stream does it keep the property. (and it will keep being notexpanded as long as it's untouched, whatever you do with the token afterwards)

Even with the current Lua engine I don't see a way to fix this one.

---

Some other interesting properties (tested with LuaTeX where the meaning is [unknown command code! (0, 1)] instead of the usual \relax)

• Even when it's skipped over with \expandafter, it also lose its \notexpanded: property.
• Similarly, when its meaning is assigned to with \futurelet it lose its \notexpanded: property; but importantly, the assignment target gets the \notexpanded meaning and it will remains so..
• If it's left on the input stream then it's set to some unexpandable token e.g. \let, then its meaning will become \let; however if it's set to something expandable after that its meaning will return to be notexpanded.
• On the other hand, if it's originally something unexpandable, made notexpanded, then redefined to be something expandable, its meaning will be notexpanded.
• Lua's token.put_next(token.get_next()) will make it lose the property as well.