2

Assume tokens
\macro{1612512}2{1612612}2{1⟨arbitrary non-outer brace balanced tokens⟩}2;

  • the 1st argument of \macro holds a TeX-⟨number⟩-quantity in the range of possible codepoints of the TeX engine's internal character encoding scheme which denotes the character code of the explicit category 1 character token to create;
  • the 2nd argument of \macro holds a TeX-⟨number⟩-quantity in the range of possible codepoints of the TeX engine's internal character encoding scheme which denotes the character code of the explicit category 2 character token to create;
  • the 3rd argument of \macro holds an arbitrary amount of non-outer brace balanced tokens that are to be placed between the explicit category 1 character token and the explicit category 2 character token.

How to define an expandable macro \macro in Knuth's TeX—\expanded not available, where expansion at some stage delivers
A1⟨arbitrary non-outer brace balanced tokens⟩B2 ?

How to define an expandable macro \macro for an (elder) utf-8-engine with \expanded not being available, where expansion at some stage delivers
A1⟨arbitrary non-outer brace balanced tokens⟩B2 ?


With \expanded/expl3 available you can do s.th. along

\ExplSyntaxOn
\cs_new:Npn \macro #1#2#3 
  {
    \tex_expanded:D { \char_generate:nn {#1}{1}
                      \exp_not:n {#3}
                      \char_generate:nn {#2}{2}
                    }
  }
\ExplSyntaxOff

If expandability was not an issue, one could do s.th. along

\ExplSyntaxOn
\newtoks\scratch
\cs_new:Npn \macro #1#2#3 
  {
    \scratch={#3}
    \exp_after:wN \def
    \exp_after:wN \test 
    \exp_after:wN {
     \exp:w
     \exp_after:wN \exp_after:wN \exp_after:wN
     \exp_after:wN
     \exp_after:wN \exp_after:wN \exp_after:wN
     \exp_end: 
     \exp_after:wN \exp_after:wN \exp_after:wN
     \exp_after:wN
     \char_generate:nn {#1}{1}
     \the \exp_after:wN \exp_after:wN \exp_after:wN \scratch \char_generate:nn {#2}{2}
    }
  }
\ExplSyntaxOff

It is academical interest. I try to learn how TeX works in order to gain a good understanding of what you can do and what you can't. And how. I think that approach is better than to start by trying to write documents and getting stuck all the time while approaching the deadline.

I know expl3 is LaTeX, but the only real expl3-thing I use is \char_generate:nn and according to source3.pdf that is fully functional with traditional LaTeX (8 bit engines) also, which implies that a stripped down variant only delivering explicit character tokens of category 1/2 can also be implemented using plain tex. E.g., one could define a macro for each code point which with two expansion-steps, \romannumeral-expansion and brace-hacking delivers the character token in question.

9
  • why the restriction to classic tex? Commented Feb 14 at 22:56
  • @DavidCarlisle Because I seem to already know how to do it using \expanded. I don't have practical use for this yet - I want to learn what's possible and what's not and what I overlook.
    – user301872
    Commented Feb 14 at 22:59
  • do you require this to be expandable ? I doubt that's possible in classic tex. Commented Feb 14 at 23:00
  • but latex for example is not usable in classic tex, pdftex isn't classic tex plain tex has etex available. so the restriction is like asking to do it in tex2 or tex-in-sail, possibly of academic interest but of no practical use Commented Feb 14 at 23:01
  • @DavidCarlisle Yes, I require expandabilitym because I seem to know hiow to do it when expandability is not an issue. ;-)
    – user301872
    Commented Feb 14 at 23:21

2 Answers 2

2

There is no reason to do this but if you define 256*256 helper functions you could do

\def\macro#1#2{%
  \csname x\number#1x\number#2\endcsname
  }


  % then lots of definitions like

  {\catcode65=1\relax \catcode66=2\relax
  \expandafter
  \gdef\csname x65x66\endcsname#1{A#1B}
}


\immediate\write20
\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter
\expandafter
{%
  \macro{65}{66}{some \relax \alpha tokens}%
}

\bye
0
3

When David Carlisle pointed out the right direction I realized that one can do with (256·2)+3 helper macros instead of 2562:

\newcount\scratch
\long\def\gobble#1{}%
\chardef\nomoreromannumeral=0 %

\long\def\PlaceCategoryTwoTokenBehindArgument#1#2{%
  \endgroup
  %---------------------------------------------------------
  \expandafter\def
  \csname PlaceACategoryOneToken\number\scratch\endcsname{%
    \expandafter\expandafter\expandafter\nomoreromannumeral
    \expandafter\expandafter\expandafter#1%
  }%
  %---------------------------------------------------------
  \long\expandafter\def
  \csname PlaceACategoryTwoToken\number\scratch BehindArgument\endcsname##1{%
    \expandafter\gobble\string#2%
  }%
}%

% Character with code-point number 0 = Character ^^@ cannot
% be lowercased,see TeXbook, so needs treatment outside
% lowercase loop:
%---------------------------------------------------------
\catcode0=1\relax
\expandafter\def
\csname PlaceACategoryOneToken0\endcsname{%
  \expandafter\expandafter\expandafter\nomoreromannumeral
  \expandafter\expandafter\expandafter^^@\expandafter\gobble\string}%
}%
%---------------------------------------------------------
\catcode0=2\relax
\long\expandafter\def
\csname PlaceACategoryTwoToken0BehindArgument\endcsname#1{%
  \expandafter\gobble\string{#1^^@%
}%
%---------------------------------------------------------
\catcode0=9\relax

\catcode`\X=1\relax
\catcode`\Y=2\relax

\scratch=0\relax
\loop
\ifnum\scratch<255 %
  \advance\scratch by 1 %
  %---------------------------------------------------------
  \begingroup
  \lccode`\X=\scratch\relax
  \lccode`\Y=\scratch\relax
  \lowercase{\PlaceCategoryTwoTokenBehindArgument{X\expandafter\gobble\string}}{{##1Y}}%
  %---------------------------------------------------------
\repeat

\catcode`\X=11\relax
\catcode`\Y=11\relax


\long\def\PlaceCategoryTwoTokenBehindArgument#1#2{%
  \csname PlaceACategoryTwoToken\number#2BehindArgument\endcsname{#1}%
}%
\def\PlaceCategoryOneToken#1{%
  \csname PlaceACategoryOneToken\number#1\endcsname
}%
\long\def\macro#1#2#3{%
  \romannumeral\PlaceCategoryTwoTokenBehindArgument{\PlaceCategoryOneToken{#1}#3}{#2}%
}%

% A test:
\newtoks\testtoks

\testtoks\expandafter\expandafter\expandafter{%
  \macro{65}{66}{\fi#\endcsname\iffalse ##\UndeFInED\egroup\par\csname\if\bgroup}%
}%
\edef\test{\the\testtoks}%
\show\test

\bye

Edit:

After Joseph Wright gave the challenge in chat I realized that one can do with 256+5 helper macros:

\newcount\scratch
\long\def\gobble#1{}%
\long\def\fot#1#2{#1}%
\long\def\sot#1#2{#2}%
\chardef\nomoreromannumeral=0 %
\chardef\MyOne=1 %

% Character with code-point number 0 = Character ^^@ cannot
% be lowercased,see TeXbook, so needs treatment outside
% lowercase loop:
%---------------------------------------------------------
\begingroup
\catcode`\^^@=1\relax
\def\fot{\expandafter\expandafter\expandafter^^@\expandafter\gobble\string}}%
\catcode`\^^@=2\relax
\def\sot{\expandafter\gobble\string{^^@}%
\edef\fot{%
  \endgroup
  \long\noexpand\expandafter\def
  \noexpand\csname PlaceACategoryOneOrTwoToken0BehindArgument\noexpand\endcsname##1##2{%
    \noexpand\ifnum##1=\nomoreromannumeral
    \noexpand\expandafter\noexpand\fot\noexpand\else\noexpand\expandafter\noexpand\sot\noexpand\fi
    {%
      \noexpand\expandafter\noexpand\gobble\noexpand\string{##2\sot
    }{%
      \noexpand\expandafter\noexpand\expandafter\noexpand\expandafter\nomoreromannumeral
      \noexpand\expandafter\noexpand\expandafter\noexpand\expandafter\fot
      \noexpand\expandafter\noexpand\gobble\noexpand\string}%
    }%
  }%
}%
\fot

%---------------------------------------------------------
\long\def\PlaceCategoryOneOrTwoTokenBehindArgument#1#2{%
  \long\expandafter\def
  \csname PlaceACategoryOneOrTwoToken\the\scratch BehindArgument\endcsname##1##2{%
    \ifnum##1=\nomoreromannumeral\expandafter\fot\else\expandafter\sot\fi
    {%
      \expandafter\gobble\string#1%
    }{%
      \expandafter\expandafter\expandafter\nomoreromannumeral
      \expandafter\expandafter\expandafter#2%
    }%
  }%
}%
%---------------------------------------------------------

\catcode`\X=1\relax
\catcode`\Y=2\relax

\scratch=0\relax
\loop
\ifnum\scratch<255 %
  \advance\scratch by 1 %
  %---------------------------------------------------------
  \begingroup
  \lccode`\X=\scratch\relax
  \lccode`\Y=\scratch\relax
  \lowercase{%
    \endgroup
    \PlaceCategoryOneOrTwoTokenBehindArgument{{##2Y}{X\expandafter\gobble\string}}%
  }%
  %---------------------------------------------------------
\repeat

\catcode`\X=11\relax
\catcode`\Y=11\relax

\long\def\PlaceCategoryOneOrTwoTokenBehindArgument#1{%
  \csname PlaceACategoryOneOrTwoToken\number#1BehindArgument\endcsname
}%

\long\def\macro#1#2#3{%
  \romannumeral\PlaceCategoryOneOrTwoTokenBehindArgument{#2}{\nomoreromannumeral}{%
    \PlaceCategoryOneOrTwoTokenBehindArgument{#1}{\MyOne}{}#3%
  }%
}%


% A test:
\newtoks\testtoks

\testtoks\expandafter\expandafter\expandafter{%
  \macro{65}{66}{\fi#\endcsname\iffalse ##\UndeFInED\egroup\par\csname\if\bgroup}%
}%
\edef\test{\the\testtoks}%
\show\test

\bye

Now I am interested if there is an expandable variant which also does on utf8-engines where you don't have \expanded. (1,114,112 code-points as in unicode would require a lot of helper macros...)

14
  • \chardef\nomoreromannumeral=`\^^00 is just \chardef\nomoreromannumeral=0 and so unlike `\^^0 the token \nomoreromannumeral has no effect on expansion of the following token Commented Feb 15 at 9:39
  • @DavidCarlisle Thanks for having a look.
    – user301872
    Commented Feb 15 at 11:18
  • @DavidCarlisle a) You are right: \chardef\nomoreromannumeral=`\^^00 indeed is a complicated way of saying \chardef\nomoreromannumeral=0 . (A ⟨normal integer⟩ of kind backtick⟨character token⟩⟨one optional space⟩ instead of <integer constant⟩⟨one optional space⟩.)
    – user301872
    Commented Feb 15 at 11:18
  • @DavidCarlisle b) But \nomoreromanumeral is ⟨chardef-token⟩. TeXbook says in chapter "24 Summary of Vertical Mode" that ⟨chardef-token⟩ is ⟨internal integer⟩ which in turn is a kind of ⟨normal integer⟩ and that unlike with other kinds of ⟨normal integer⟩/⟨internal integer⟩ with this kind of ⟨internal integer⟩ no trailing ⟨one optional space⟩ is searched. So when \romannumeral - after doing the expansion you want - finds non-positive ⟨internal integer⟩, then you don't get any token in return and TeX does not keep on expanding things for finding components of \romannumeral's ⟨number⟩.
    – user301872
    Commented Feb 15 at 11:18
  • @DavidCarlisle So there are subtle differences which let me prefer ⟨chardef token⟩ for ending \romannumeral-expansion:
    – user301872
    Commented Feb 15 at 11:18

You must log in to answer this question.