Mainly for debugging purposes I would like to have a macro which prints all tokens in a list of tokens and their corresponding catcodes.
Based on an answer by Joseph Wright I have written a macro which returns the catcode of one token.
Based on this question which shows three different loops I see two fundamentally different approaches two iterate over the tokens:
dealing with explicit tokens by passing them as arguments or dealing with implicit tokens using \let
. The latter seems more promissing therefore I am calling it approach 1.
I am having the following problems:
- using approach 1 (implicit tokens):
- I don't know how to print the symbol of an implicit token if it does not have catcode 11 or 12 (letter or other).
- Active characters are printed as catcode 16.
- using approach 2 (explicit tokens):
- Spaces are gobbled.
{
and}
are not printed but used to make groups.
I don't think that the problems of the second approach can be solved. But I am hoping that there are improvements for the first approach or that there might be a third, better approach which I am missing.
\documentclass{article}
\usepackage{filecontents}
% ========== get catcode ==========
% I am not using something like \the\catcode`#1 because:
% (1) I want the catcode of the token,
% not the catcode which a token would get
% if it was created at this position
% (2) that would not work with implicit tokens
\makeatletter
\@firstofone{\let\implicitSpaceToken= }
\makeatother
% based on https://tex.stackexchange.com/a/7413/120953
\newcommand{\getCatcode}[1]{%
% 0: escape character, no tokens of that catcode exist
\ifcat \egroup\noexpand#1%
1%
\else\ifcat \bgroup\noexpand#1%
2%
\else\ifcat $\noexpand#1%$ (the commented out dollar sign is important for the syntax highlighting in TeXstudio)
3%
\else\ifcat &\noexpand#1%
4%
% 5: end of line, no tokens of that catcode exist
\else\ifcat ##\noexpand#1%
6%
\else\ifcat ^\noexpand#1%
7%
\else\ifcat _\noexpand#1%
8%
% 9: ignored character, no tokens of that catcode exist
\else\ifcat \implicitSpaceToken\noexpand#1%
10%
\else\ifcat a\noexpand#1%
11%
\else\ifcat 1\noexpand#1%
12%
\else\ifcat \noexpand~\noexpand#1%
13%
% 14: comment character, no tokens of that catcode exist
% 15: invalid character, no tokens of that catcode exist
\else\ifcat \relax\noexpand#1%
16%
\else
error% this can not happen
\fi \fi \fi \fi \fi \fi \fi \fi \fi \fi \fi \fi
}
% ========== approach 1: loop based on implicit tokens ==========
% + does not ignore spaces
% + no problems with groups
% - wrong catcode for active characters (16 instead of 13)
% - I don't see a feasable way to print the characters (other than those with catcode 10 and 11)
\begin{filecontents}{loop-implicit-tokens.tex}
\def\printtokens#1{
% \def\do##1{$\texttt{\string##1}_{\getCatcode##1}$}%
\def\do##1{%
\edef\i{\getCatcode{##1}}%
(\i%
\ifnum \i = 11\relax
:\,\texttt{##1}%
\else\ifnum \i = 12\relax
:\,\texttt{##1}%
\fi \fi
)%
}%
\iterate#1\relax
}
% based on https://tex.stackexchange.com/q/359189/120953
\def\iterate{\afterassignment\loopbody\let\xchar= }
\def\loopbody{%
\ifx\relax\xchar
\let\next=\relax
\else
\do\xchar
\let\next=\iterate
\fi
\next
}
\end{filecontents}
% ========== approach 2: loop based on explicit tokens ==========
% + prints correct catcode of active characters
% + possible to print the character
% - ignores *explicit* tokens with catcode 10 (space)
% - problems with *explicit* tokens of catcodes 1 and 2 (groups)
\begin{filecontents*}{loop-explicit-tokens.tex}
\def\printtokens#1{
\def\do##1{$\texttt{\string##1}_{\getCatcode##1}$}%
\iterate#1\relax
}
% based on https://tex.stackexchange.com/q/359189/120953
\def\iterate#1{%
\ifx\relax#1%
\else
\do{#1}%
\expandafter\iterate
\fi
}
\end{filecontents*}
% ========== main document ==========
\input{loop-implicit-tokens}
%\input{loop-explicit-tokens}
\newcommand{\printtokensinmacro}[1]{\expandafter\printtokens\expandafter{#1}}
\makeatletter
\newcommand{\test}{@ $i_\text{di}^2$&##~}
\makeatother
\begin{document}
\printtokensinmacro\test
\end{document}
\peek_N_type:TF
from expl3 you are almost done, check if N type, grab as an argument and process, if not, check if it's a space, or a brace and output the correct catcode for those three particular cases.