Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [token-lists]

The tag has no usage guidance.

4
votes
2answers
85 views

Convert token to string in LaTeX3

While trying to answer another question on this site, I was experimenting with \peek_after:Nw and \l_peek_token. I can set \l_peek_token OK, but then I want to test if this token is one of several ...
3
votes
1answer
45 views

How to pop (right) a token list with expl3?

The expl3 package provides \tl_put_right which enables us to append tokens to the right side of the current content of a token list variable: \documentclass{article} \usepackage{xparse} \begin{...
1
vote
0answers
29 views

Store tokens with unbalanced braces in token register

First of all, this question is not part of a real-world problem, it's just to see whether the following is possible with TeX or not. Consider this short code: \begingroup \toks0={\def\foo<} \...
2
votes
5answers
161 views

How can I condition on the last token?

I would like to write a command \foo which, for the sake of simplicity, does just one thing, which is make sure its argument ends with a dot. Thus both foo{hello world} and foo{hello world.} expand to ...
2
votes
1answer
70 views

expl3 read a comma list from file

I am using expl3 now, I want to read data from file, and store them in a comma list, then use the comma list one by one. I don't have any code, I just know exp13 have these function, so need someone ...
3
votes
1answer
54 views

Latex3: multicolumn at the beginning of a token list produces an error

I'm not sure if using token list is a good approach to generating table data. Anyway, here is a weird result. It works well in all other cases except when the first item in token list is multicolumn. ...
1
vote
1answer
31 views

Generate a list of paragraph and page number for specific labels

I was wondering if anyone could help me with this: I've got this report for which I show the paragraph numbers. The paragraph number is then used as part of the label so I can extract it later on, ...
6
votes
1answer
99 views

How can I implement a \futurelet that ignores spaces?

This is a question about token processing in TeX. What am I doing wrong when implementing \FutureLetNoSpace? The book TEX in Practice: Volume III: Tokens, Macros, by Stephan v. Bechtolsheim, really ...
5
votes
1answer
92 views

expl3 — sort a token list

L A TEX3 comes with a facility to sort list variables (sequences, token lists, or comma-lists) according to some user-defined comparison. I try to adapt the interface3 manuel example to a token list, ...
2
votes
1answer
67 views

Building up and joining “internal” lists

How can I append to a pgfkeys-style comma-separated list? Or, alternatively, how can I define a macro to join an etoolbox list with a delimiter? For an environment I'm designing, I want to accumulate ...
1
vote
1answer
63 views

LaTeX output local lists on root document

This is a followup question for this question, where Christian Hupfer provided me a great solution for a certain problem: I wanted to list the occurrences of special commands within a section at the ...
2
votes
2answers
95 views

Why “Missing Control Sequence Inserted” using Pop Function on Token Register?

When using the \Pop macro from from TeX by Topic chapter "Operations on token lists: stack macros", I get Missing control sequence inserted. Why? I copied it exactly as is! Was there a typographical ...
5
votes
2answers
151 views

Latex3 Token-List commands do not work as expected

Let's take a trivial case: getting the head and tail of a token list and reversing the order: \documentclass{article} \usepackage{xparse} \usepackage{expl3} \ExplSyntaxOn \cs_set:Npn \cs_split:n #...
6
votes
1answer
116 views

How can I define a macro with the value of a fully expanded macro?

Related Expansion of macros and box contents Why isn't a command defined by \newcommand with an optional argument expandable? I made some simple macros for lists, but I do not know how ...
3
votes
2answers
63 views

Applying Macro to xth Symbol of String

I would like to define a command that puts a sequence of symbols above the xth symbol of another sequence of symbols. It seems nicer without a separator, but for higher usability a separator may be ...
1
vote
1answer
179 views

How to create a local token list variable in LaTeX3?

The \tl_new:(Nc) functions create a new token list variable globally. Is there a way to create a new token list variable locally?
2
votes
4answers
235 views

How to handle a LaTeX3 token list as a list of tokens rather than as a list of items?

Is there a way to query and manipulate a LaTeX3 token list as a list of tokens rather than as a list of items? For instance, given the token list {a}b is it possible to find out how many tokens it ...
1
vote
2answers
116 views

Q: Loop over all tokens in the body of a macro

I am looking for TeX-code that loops over all tokens in a macro definition. \meaning transforms everything to "other" (catcode 12) characters and is thus not directly suited (with or without \@strip@...
5
votes
1answer
285 views

What are the constituents of a LaTeX3 token list?

I'm trying to understand what a LaTeX3 token list is. Unfortunately, the LaTeX3 documentation is anything but helpful. According to the LaTeX3 Interfaces documentation token lists can be viewed in two ...
5
votes
1answer
207 views

TeX's expansion rules: a case study with a token-list register

I'm trying to figure out TeX's rules of expansion. Consider the following plain TeX manuscript: \toks1{hello}% \toks10{world}% \def\mac{0}% \the\toks1\noexpand\mac% \bye When this manuscript is ...
1
vote
2answers
103 views

Is there a better method for deferring the execution of commands?

The problem I am interested in getting the following result, by not writing it directly in code. \begin{align} T(x) &= ae^{-\frac{(x-b)^2}{2c^2}} \\ \frac{1}{2} < a &\leq 1 \label{...
0
votes
0answers
42 views

Are using lists to store commands prone to stack exhaustion?

Background Using a handful of answers in this site, I have come up with a solution for building a list of equations (with tocloft), such that two or more of them can be grouped inside an align ...
1
vote
1answer
160 views

'Runaway definition' error when a token list variable is defined inside a macro

The following plain TeX manuscript \def\mymacro{\newtoks\mytoks}\bye fails to compile and generates the following error: Runaway definition? -> ! Forbidden control sequence found while scanning ...
1
vote
1answer
40 views

Assigning to a token list register without a `=` sign

I've noticed that it is possible to assign a value to a token list register without using = (the equality sign) as follows: \newtoks\mytoks% \mytoks{Hello, world!}% as opposed to \mytoks={Hello, ...
4
votes
1answer
87 views

How to create list and assign it to variable?

I need to create a comma (or any other) separated list of words for further usage and assign it to variable (macro). The words in list should be placed in separate lines. When I use for this purposes \...
3
votes
2answers
52 views

Macro for replacing matching tokens in a parameter?

I would like to have a macro, call it \magicreplace which gets some arbitrary parameter (= sequence of tokens) and replaces every occurrence of some token (say, a or x or ,) with some fixed sequence ...
2
votes
1answer
175 views

How are the operations of the TeX tokenizer and engine interleaved?

On p. 38 of Knuth's The TeXbook (1996), a TeX system is broken down into two components: the "mouth" that "chews" the .tex manuscript and converts it to tokens, and the "stomach" that does the actual ...
4
votes
1answer
118 views

TeX's reading rules (text to tokens)

Are TeX's reading rules, which define the conversion from text to tokens, and which are described on pp. 46-48 of Knuth's The TeXbook (1996), describe correctly and completely the reading rules of the ...
6
votes
3answers
162 views

Populating tabular with etoolbox list

I am trying to populate a tabular environment with an etoolbox list: \documentclass{article} \usepackage{etoolbox} \listadd\mylist{Ana}% \listadd\mylist{Bob}% \listadd\mylist{Cole}% \newcommand*{\...
6
votes
2answers
368 views

Can I test line break types? e.g. line endings of type LF or LFCR?

Is it possible to test whether a "line feed" or "line feed + carriage return" were used to end lines in a tex file? I'd like to create a log message with this information. The reasoning for this is ...
2
votes
1answer
63 views

Why does \tl_if_in (and \clist_if_in) insert extra tokens with \mathrm

I encountered a strange behaviour I can’t explain. In my code for a highlighting macro I tried to use \tl_if_in to do a basic math class test. But when \mathrm is part of the tokens to be tested some ...
2
votes
2answers
61 views

Inverted token list in a table

I read a lot of questions and answers about this matter, each doing a little of what I was looking for, but when I tried to merge them together, the whole world collapsed... My goal is to build a ...
1
vote
1answer
74 views

Trying to create dynamic table and commands alltogether

I'm trying to do a command that allow the user to create a specific table for a document but that the person can decide on the fly on the name of the tables (so he can dynamically create multiple ...
2
votes
2answers
100 views

expl3, incrementing a value in a prop-list

I read a clist and create a prop-list which should behave as a hash in which the value for a given key is the number of occurences of this value in the clist. Below my code which, so far, does work as ...
8
votes
2answers
92 views

How to store to-be-credited contributors for acknowledgment later in the appendix

In my thesis, I create numerous drawings using TikZ. In these drawings, I include pictures (e.g. icons of The Noun Project) using Creative Commons licenses. I thus have to acknowledge their authors. ...
15
votes
3answers
714 views

Character bytes and character tokens: If newlines are converted to spaces, then where does catcode 5 come into the picture?

My goal is to understand the relationship between character bytes and character tokens with respect to new line bytes. I likely do not have my facts straight. When TeX reads a file of bytes, encoding ...
14
votes
3answers
642 views

Why does the TeX scanner process tokens for register numbers and macro names differently?

This is a "How does TeX know when to stop" question. Case 1 Note: I have since found out that my understanding here is wrong. See SPOILER and answers. When TeX is constructing a macro name, it ...
2
votes
1answer
118 views

How to define \pgfkeyssetexpandedvalue?

Note: this question follows my previous question about \pgfkeyssetvalue. The \pgfkeyssetvalue is defined in pgfkeys.code.tex file by: \newtoks\pgfkeys@temptoks % [...] \long\def\pgfkeyssetvalue#1#2{...
3
votes
1answer
89 views

Why \pgfkeyssetvalue uses a token list?

The \pgfkeyssetvalue is defined by (in pgfkeys.code.tex file): \long\def\pgfkeyssetvalue#1#2{% \pgfkeys@temptoks{#2}\expandafter\edef\csname pgfk@#1\endcsname{\the\pgfkeys@temptoks}% } where \...
0
votes
1answer
198 views

Beamer boxes with variable sizes and position in frame

I want to to build flexible beamer boxes so that I can change the width and alignment within the frame. Passing the options width= and alignment=, whereas width is a valid value and alignment can be ...
4
votes
1answer
276 views

Obtaining the name of an undefined control sequence in LuaTeX

I want to implement something similar to \detokenize as a Lua function. But there is one case where I am prevented from accomplishing my goal. \directlua{tex.sprint("[" .. token.get_next().csname .. "...
12
votes
2answers
515 views

How are parameter tokens (#1,#2,…,#9) processed?

I wonder how the TeX token muncher processes parameter arguments. It seems that parameter arguments are sent token by token. Is this true? Consider the following code based on wipet's answer used to ...
4
votes
2answers
123 views

\tl_gput_right:Nn gets ! TeX capacity exceeded, sorry [input stack size=5000]

The following file gets ! TeX capacity exceeded, sorry [input stack size=5000]. If I remove the call to \tl_gput_right:Nn then it compiles successfully. \documentclass{article} \usepackage{amsfonts} \...
14
votes
1answer
151 views

Avoid @ hackery: How can I preserve spaces in a format specification when processing the specification with expl3 and PGF/TikZ?

This is a simplified version of part of the code I've been using in answers based on chronos, an attempt to create something somewhat like a substitute for the problematic chronosys. This code works ...
3
votes
2answers
75 views

Using token lists in a newenvironment definition

I'm trying to define a new environment to format some user scenarios for a project I'm working on. The MWE below shows what's happening. It appears that the first use of \\ empties the token lists. ...
4
votes
1answer
109 views

Is it possible to inject a token after another token?

Something like: If next token is some token that artificially creates a page break (e.g. \newpage, \clearpage Then inject something after it Else inject something before it I want to modify \input ...
5
votes
2answers
367 views

Why does \detokenize # sometimes fail inside of environments?

When I use the pound sign/number sign/hash sign # inside of commands and environments, this refers to arguments. However, shouldn't \detokenize{#} make it NOT an argument? Situation The pound sign ...
1
vote
0answers
50 views

Need help with grouping and expansion control

I am trying to define macros, which take a delimited list and pass each element to a macro (that takes one argument). I have fully working versions of them that iterate over 1) all elements in a list, ...
4
votes
2answers
81 views

How to process only tokens containing a lower case letter?

The goal is to have a macro with single argument, that looks at all tokens in turn and if it sees a lower-case letter it is wrapped in \texttt{<token>}. All other tokens are passed through ...
4
votes
1answer
71 views

Fake math sizing using tokens assignment

I'd like to implement sizing functions like \Huge in a way that makes them work in math mode as well. Essentially I want to create a command myHuge which maps {foo \myHuge bar} to {foo \mbox{\Huge $...