Reconstructing a macro from its \meaning in tex-core

Question

In order to gain a deeper understanding of TeX I am following this guide on TeX Programming. It contains examples on macro expansion like this one

\def\point(#1,#2){we do something with #1 and #2}
\def\temp{(42,1234)}
\expandafter\point\temp

which got me tinkering with tring to reconstruct a macro from it's definition.

Here is my approach:

% definition of macro
\def\macro #1{I am a macro called with #1}
% storing it in toks1
\toks1={\meaning\macro}

% define reconstruction of arguments/pattern from \meaning
\def\args #1:#2->#3{#2}%
% define reconstruction of definition from \meaning
\def\definition #1:#2->#3{#3}%

% define reconstruction itself
\def\reconstruct(#1,#2){\expandafter\def\csname#1\endcsname \args #2 {\definition #2}}%

% reconstruct contents of toks1
\reconstruct(reconstructed,\expandafter\expandafter\null\the\toks1)
% print its meaning
\message{\meaning\reconstructed}


\end

this however does not result in the macro that I want to be reconstructed. I can't use it in the intended way, nor does it's \meaning mirror the original macro.

I tried various different placements for \expandafter, but this was ultimately the most logical way to do it (for me at least).

what is the proper way to do this using TeX only?

I think that the expansion as done above will result in something like this

\def\mymacoro #1{\def\mymacro2 #1{#1}}

wich is obvioulsy a problem; can I help myself with a group there or are # registers not shadowed?

Answer 1

There are several problems with you approach. You should first store the meaning, not \meaning\macro, so

\toks0=\expandafter{\meaning\macro}

(scratch registers with odd number should be reserved for global assignments, but it's a minor point). Now, \showthe\toks0 will print

> macro:#1->I am a macro called with #1.

but you'll never be able to reconstruct \macro from this data unless you write the result in a file and read it or, equivalently, use \scantokens that's only available with e-TeX engines (so not with Knuth TeX).

The problem is that the result of \meaning\macro is composed of category 12 character tokens (but spaces have category code 10).

This works for macros with undelimited arguments (with limitations on the replacement texts, which must not contain \replacement).

It might also work with delimited arguments.

\newwrite\reconstructout
\newread\reconstructin

\def\macro #1{I am a macro called with #1}

\immediate\openout\reconstructout=\jobname.rcn
\immediate\write\reconstructout{\meaning\macro}
\immediate\closeout\reconstructout
\openin\reconstructin=\jobname.rcn
\chardef\remember=\endlinechar
\endlinechar=-1
\read\reconstructin to \reconstruct
\closein\reconstructin
\endlinechar=\remember

\def\args #1:#2->#3\args{#2}
\def\replacement #1:#2->#3\replacement{#3}

\toks0=\expandafter\expandafter\expandafter{\expandafter\args\reconstruct\args}
\toks2=\expandafter\expandafter\expandafter{\expandafter\replacement\reconstruct\replacement}

\begingroup\edef\x{\endgroup
  \def\noexpand\mymacro\the\toks0{\the\toks2}%
}\x

\ifx\mymacro\macro\message{SUCCESS!!!}\fi

Here's the console output:

This is TeX, Version 3.141592653 (TeX Live 2021) (preloaded format=tex)
(./reconstruct.tex SUCCESS!!!)

It also works if you do

\def\point(#1,#2){we do something with #1 and #2}

and replace \macro in the code above with \point.

Answer 2

\def\macro #1{I am a macro called with #1}
% storing it in toks1
\toks1={\meaning\macro}

at this point toks1 contains the two tokens \meaning\macro I think you wanted it to hold the string from \meaning which would be

\toks1=\expandafter{\meaning\macro}

Note however that \meaning is a "verbatim" string everything is catcode 12 (punctuation) or 10 (space) so in particular the letters in the text are not catcode 11 (letter) and the # in #1 is a normal puctuation # like \# not the macro parameter.

In classical TeX to reparse this string assuming normal catcodes you could write it out to a file and then read it back, e-tex offers \scantokens to reparse (internally it does more or less exactly the same as a file write followed by a file read).

Similarly

\expandafter\expandafter\null\the\toks1

The first \expandafter expands \null so

\expandafter\hbox{}\the\toks1

so the second \expandafter expands { which is not expandable so this is

\hbox{}\the\toks1

which is not what you intended. (It isn't clear what you wanted to do with \null in any case it adds a box which is not wanted)

Answer 3

For gaining deeper understanding of TeX I recommend reading the TeXbook, being very picky about the context of every word while reading and strictly sticking to correctly using the terminology introduced in that book.

The guide of programming referenced by you uses its own home-brewed terminology which lacks of precision and therefore may be misleading for novices.

E.g., you find explanations like

But neither is explained what "contents" in the phrase "contents of a macro" means, nor is explained what "as it is seen by TeX" means.

Besides this, applying \meaning actually is not restricted to macros. \meaning can be applied to a ⟨token⟩ which is not a ⟨macro⟩ as well: You can use \meaning for finding out about the meanings of primitives, explicit character-tokens, implicit character-tokens, \chardef-tokens, \toksdef-tokens, \dimendef -tokens, whatsoever \...def-tokens, ...

A novice cannot deduce that characters occurring in the .tex-input-file may have been discarded at the time of tokenizing the definition of the macro whose \meaning is to be delivered because at that time those characters were of category code 10(space) and thus got discarded in moments where the reading-apparatus was in state S(skipping blanks) or N(new line).

A novice cannot deduce that \meaning just delivers a sequence of explicit character-tokens of category 12(other) with spaces being the only exception: Spaces will be of category 10(space). I.e., \meaning does not give you clues which of the character-tokens it delivers come from control-word-tokens or control-symbol-tokens and which of the character-tokens it delivers come from explicit character-tokens. Besides this, with explicit character-tokens you loose information about categories. Routines for patching macros that are based on \meaning actually just do some sort of "very educated guessing".

A novice cannot deduce that the result of \meaning is affected by the value of the integer-parameter \escapechar.

A novice cannot deduce that the result of \meaning may be affected by category codes that are current at the time of carrying out \meaning: When \meaning delivers the symbolic representation of a control-symbol-token, no space-token is appended. When \meaning delivers the symbolic representation of a control-word-token, a space-token is appended. Whether a control-sequence-token whose name consists of a single character is treated as a control-symbol-token or as control-word-token depends on the category-code of that character current at the time of carrying out \meaning:

\def\test{\z\z W\z}
\tt
\string\test=\meaning\test

\catcode`\z=12
\string\test=\meaning\test

\escapechar=`\A
\catcode`\z=11
\string\test=\meaning\test

\catcode`\z=12
\string\test=\meaning\test

\bigskip
\hrule
\bigskip

\escapechar=`\\
\catcode`\z=11
\def\testA{\z\zW \z}
\def\testB{\z\z W \z}
\catcode`\z=12
\string\testA=\meaning\testA

\string\testB=\meaning\testB

meanings are \ifx\testA\testB equal\else different\fi

\bye

TeXbook, Chapter 7: How TeX Reads What You Type clearly says:

In the examples so far, \string has converted control sequences into lists of tokens that begin with \12 . But this backslash token isn't really hardwired into TeX; there’s a parameter called \escapechar that specifies what character should be used when control sequences are output as text. The value of \escapechar is normally TeX’s internal code for backslash, but it can be changed if another convention is desired.

This paragraph is a nice example on why I suggest being very picky while reading the book: Although in the beginning the paragraph is about \string it is not said that \string is the only routine where outputting control sequences as text plays a rôle.
\meaning, too, is such a routine.

TeXbook, Chapter 20: Definitions (also called Macros) clearly says:

\meaning⟨token⟩. TeX expands this to the sequence of characters that would be displayed on your terminal by the commands ‘\let\test=⟨token⟩ \show\test’. For example, ‘\meaning A’ usually expands to ‘the letter A’; ‘\meaning\A’ after ‘\def\A#1B{\C}’ expands to ‘macro:#1B->\C ’.
[...]
In all of the cases listed so far, \the produces a result that is a sequence of ASCII character tokens. Category code 12 (“other”) is assigned to each token, except that character code 32 gets category 10 (“space”). The same rule is used to assign category codes to the tokens produced by \number, \romannumeral, \string, \meaning, \jobname, and \fontname.

TeXbook, Chapter 21: Making Boxes clearly says:

Each \write command produces output in the form that TeX always uses to display token lists symbolically: Characters represent themselves (except that you get duplicated characters like ## for macro parameter characters); unexpandable control sequence tokens produce their names, preceded by the \escapechar and followed by a space (unless the name is an active character or a control sequence formed from a single nonletter).

Another example where pickiness while reading is a good thing: Although in the beginning the paragraph is about \write it is not said that \write is the only routine where displaying token lists symbolically plays a rôle.
\meaning, too, is such a routine.
\meaning, however, does not duplicate macro-parameter-characters.
Perhaps the word "except" is intended to indicate that this duplication is a deviation of the form of displaying token-lists symbolically which occurs when using \write for writing things (to text-file or terminal).