Questions tagged [token-lists]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
22
votes
5answers
2k views

Basics of parsing

I'm trying to master the art of parsing text one character at a time. The following is what I've discovered so far. Is this as good as it gets? I would really prefer not to do all those \edefs as ...
9
votes
3answers
364 views

Appending active and expansion sensitive characters to a macro

This question comes up pretty often when people want to build tables with empty columns or similarly building up some active characters inside a macro hoping to expand them at the right time right ...
72
votes
4answers
5k views

What is a token?

Please forgive such a basic question, but I did some Google searching and looked around on this site, and while there are many threads and online resources that involve tokens, I can't seem to find a ...
20
votes
2answers
370 views

Expandable test for an empty token list—methods, performance, and robustness

With ε-TeX, the go-to method for testing if a <token-list> is empty is the following test: \if\relax\detokenize{<token-list>}\relax % empty \else % not empty \fi The method ...
30
votes
5answers
3k views

How does one append material to a token list?

Several times, the floatrow package uses \newtoks which (according to this website) means that a token register is allocated. An example code snippet is \@ifdefinable\FR@everyfloat{\newtoks\FR@...
10
votes
2answers
2k views

Iterate over space-separated list

I’ll briefly explain the background in case this can be solved simpler but I’d still be interested in an answer to the actual question, too. My chapter titles are very narrow and use a large font so ...
19
votes
4answers
960 views

How does one remove material from a token list?

This is a follow-up question to How does one append material to a token list? The macros \prependto and \appendto work nicely to add content to token lists. How would one go about creating macros ...
10
votes
2answers
2k views

\newtoks token seems to have local scope

I want to collect the first line of a quotation to use later. I have used \newtoks for a similar purpose so tried it here, this way: \documentclass[12pt] {article} \newtoks{\qfirstlineinternal} \...
31
votes
6answers
9k views

Test whether token list is empty

Is it possible to test whether a token list register is empty without expanding it?
15
votes
1answer
165 views

Avoid @ hackery: How can I preserve spaces in a format specification when processing the specification with expl3 and PGF/TikZ?

This is a simplified version of part of the code I've been using in answers based on chronos, an attempt to create something somewhat like a substitute for the problematic chronosys. This code works ...
11
votes
3answers
554 views

Making register declarations local, Undefining a register (count, toks, etc.)

Is it possible to make \newcount, \newtoks, etc. respect grouping, so that, e.g., \newcount\temp\temp=2{\newcount\temp\temp=1}\showthe\temp gives 2 (instead of 0)? Relatedly, is it possible to ...
6
votes
2answers
149 views

How to use \cs_generate_variant:Nn to compare token lists?

I want to deal with this question by my method. The example below is some codes from interface3.pdf; But throws the error:  ! LaTeX3 Error: Variant form 'VV' deprecated for base form '\...
16
votes
3answers
793 views

Will LaTeX3 have proper support for hooks? Does LuaTeX?

Tex has powerful support for extensions triggered by events happen in generating output, by running one of six token lists appropriately: \everycr \everydisplay \everyhbox \everymath \everypar \...
20
votes
1answer
3k views

Using \toks registers

I was reading this tutorial for TeX programming: http://pgfplots.sourceforge.net/TeX-programming-notes.pdf recommended in the 2-nd answer here: Where do I start LaTeX programming? It says: \toks&...
11
votes
4answers
673 views

Improve list parsing command

This question led to part of a new package: menukeys I build a command to iterate over a list with changable seperators and it works like expected but I wonder if there is a faster or more ...
5
votes
2answers
509 views

Splitting delimited token list argument

I wrote a little piece of code (function \getfirst:w) that splits its delimited argument at the first embedded :. The code works well if used standalone, but I am unable to place it into the .code:n ...
3
votes
2answers
1k views

Extract first letter of each word in a sentence

I don't know if already exists a question like this, but after 20 minutes I haven't found one. There are similar ones, like How to define a command that bolds the first letter of each word in the ...
15
votes
3answers
860 views

Character bytes and character tokens: If newlines are converted to spaces, then where does catcode 5 come into the picture?

My goal is to understand the relationship between character bytes and character tokens with respect to new line bytes. I likely do not have my facts straight. When TeX reads a file of bytes, encoding ...
9
votes
1answer
662 views

Using \typeout behaviour different with \newtoks vs. \def

I feel as I really should know this, but sadly I don't. Why is the output of \typeout{\the\myToken} not the same as having it printed in the PDF as \the\myToken. No problem with a \def, just with ...
7
votes
4answers
1k views

Search and replace in a verbatim token list

I maintain a package for the R programming language that contains a vignette which gets processed using the Sweave literate programming system. Now, one of the irks about Sweave is that it always ...
6
votes
2answers
1k views

Passing multiple arguments from ProcessList (xparse) to a macro. Result to be used in tabular

I am making a macro that will take user input of a grading scheme and place it into a tabular environment. I would like the syntax to be as simple as possible and decided on: \tabulate{ eval1,%1 ; ...
1
vote
1answer
90 views

Trying to create dynamic table and commands alltogether

I'm trying to do a command that allow the user to create a specific table for a document but that the person can decide on the fly on the name of the tables (so he can dynamically create multiple ...
10
votes
2answers
151 views

Understanding Implicit Delimiters/Terminators

[I encourage you to check out Phelype's impressive approach] I hear about macros that do things like "expand until they reach an unexpandable token" and this question may (or may not) be related to ...
9
votes
1answer
118 views

Extra space in cells when using token lists to build tabular content

What part of my code is causing the misalignment of the first column? The extra space seems to increase with the number of rows. (Part of my code is from How do I use the ampersand (&) inside a ...
8
votes
2answers
242 views

How to count the frequency of a token in a token list

Is it possible to count the number of a token in a token list? \documentclass{article} \usepackage{expl3} \ExplSyntaxOn \tl_new:N \tobi_list_tl \tl_set:Nn \tobi_list_tl {foo~bar~baz} \begin{...
6
votes
1answer
935 views

what \relax command means?

I'm trying to compile this .tex file from ArXiVe.org in WinEdt 9.0 but some tokens are underlined with a solid line and preceeded by a strange command \relax command in WVP.aux: \relax \citation{...
6
votes
2answers
755 views

How to expand a LaTeX counter when adding to token list

In short: I am trying to append counter values to a token list, but when I print the token list at the end I get the final values of those counters. What I want is the value of the counters at the ...
5
votes
2answers
212 views

Macro for a code translator

Is there an existing macro that "translate" tokens using some code ? For example : \def\code{{{a,b},{b,c},{c,a}}} \translate[\code]{caac} % => would typeset "abba" or even better something like ...
5
votes
2answers
216 views

Using Expl3 token-list variables where token lists are called for

The code below illustrates the trouble I’m having. It seems I cannot directly use a token-list variable where a token list is called for, but why, and how can I get the result I want? \documentclass{...
4
votes
3answers
138 views

How to iterate through a token list to make characters uppercase, while preserving spaces?

The following (classical, I assume) "iterator" definition would be examined here: \def\zEnd{\zEnd} \def\zzIterator#1{% \ifx#1\zEnd \else #1% \expandafter\zzIterator \fi } \...
4
votes
2answers
212 views

Convert token to string in LaTeX3

While trying to answer another question on this site, I was experimenting with \peek_after:Nw and \l_peek_token. I can set \l_peek_token OK, but then I want to test if this token is one of several ...
4
votes
2answers
104 views

How to process only tokens containing a lower case letter?

The goal is to have a macro with single argument, that looks at all tokens in turn and if it sees a lower-case letter it is wrapped in \texttt{<token>}. All other tokens are passed through ...
3
votes
1answer
125 views

Why \pgfkeyssetvalue uses a token list?

The \pgfkeyssetvalue is defined by (in pgfkeys.code.tex file): \long\def\pgfkeyssetvalue#1#2{% \pgfkeys@temptoks{#2}\expandafter\edef\csname pgfk@#1\endcsname{\the\pgfkeys@temptoks}% } where \...
3
votes
3answers
137 views

Expl3' tl variable storing the result of a \seq_map_inline: OK in article's title but not in beamer's title

As pointed out by the following MCE, an expl3' token list variable storing the result of a \seq_map_inline: used in \title's argument: works like a charm with the article class, doesn't work with the ...
1
vote
1answer
240 views

How to create a local token list variable in LaTeX3?

The \tl_new:(Nc) functions create a new token list variable globally. Is there a way to create a new token list variable locally?
1
vote
2answers
130 views

Is there a better method for deferring the execution of commands?

The problem I am interested in getting the following result, by not writing it directly in code. \begin{align} T(x) &= ae^{-\frac{(x-b)^2}{2c^2}} \\ \frac{1}{2} < a &\leq 1 \label{...
1
vote
2answers
142 views

Q: Loop over all tokens in the body of a macro

I am looking for TeX-code that loops over all tokens in a macro definition. \meaning transforms everything to "other" (catcode 12) characters and is thus not directly suited (with or without \@strip@...
0
votes
0answers
55 views

Are using lists to store commands prone to stack exhaustion?

Background Using a handful of answers in this site, I have come up with a solution for building a list of equations (with tocloft), such that two or more of them can be grouped inside an align ...