Questions tagged [token-lists]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2
votes
1answer
231 views

How are the operations of the TeX tokenizer and engine interleaved?

On p. 38 of Knuth's The TeXbook (1996), a TeX system is broken down into two components: the "mouth" that "chews" the .tex manuscript and converts it to tokens, and the "stomach" that does the actual ...
4
votes
1answer
126 views

TeX's reading rules (text to tokens)

Are TeX's reading rules, which define the conversion from text to tokens, and which are described on pp. 46-48 of Knuth's The TeXbook (1996), describe correctly and completely the reading rules of the ...
8
votes
3answers
390 views

Populating tabular with etoolbox list

I am trying to populate a tabular environment with an etoolbox list: \documentclass{article} \usepackage{etoolbox} \listadd\mylist{Ana}% \listadd\mylist{Bob}% \listadd\mylist{Cole}% \newcommand*{\...
6
votes
2answers
465 views

Can I test line break types? e.g. line endings of type LF or LFCR?

Is it possible to test whether a "line feed" or "line feed + carriage return" were used to end lines in a tex file? I'd like to create a log message with this information. The reasoning for this is ...
2
votes
1answer
80 views

Why does \tl_if_in (and \clist_if_in) insert extra tokens with \mathrm

I encountered a strange behaviour I can’t explain. In my code for a highlighting macro I tried to use \tl_if_in to do a basic math class test. But when \mathrm is part of the tokens to be tested some ...
3
votes
2answers
75 views

Inverted token list in a table

I read a lot of questions and answers about this matter, each doing a little of what I was looking for, but when I tried to merge them together, the whole world collapsed... My goal is to build a ...
1
vote
1answer
90 views

Trying to create dynamic table and commands alltogether

I'm trying to do a command that allow the user to create a specific table for a document but that the person can decide on the fly on the name of the tables (so he can dynamically create multiple ...
2
votes
2answers
135 views

expl3, incrementing a value in a prop-list

I read a clist and create a prop-list which should behave as a hash in which the value for a given key is the number of occurences of this value in the clist. Below my code which, so far, does work as ...
8
votes
2answers
124 views

How to store to-be-credited contributors for acknowledgment later in the appendix

In my thesis, I create numerous drawings using TikZ. In these drawings, I include pictures (e.g. icons of The Noun Project) using Creative Commons licenses. I thus have to acknowledge their authors. ...
15
votes
3answers
860 views

Character bytes and character tokens: If newlines are converted to spaces, then where does catcode 5 come into the picture?

My goal is to understand the relationship between character bytes and character tokens with respect to new line bytes. I likely do not have my facts straight. When TeX reads a file of bytes, encoding ...
14
votes
3answers
741 views

Why does the TeX scanner process tokens for register numbers and macro names differently?

This is a "How does TeX know when to stop" question. Case 1 Note: I have since found out that my understanding here is wrong. See SPOILER and answers. When TeX is constructing a macro name, ...
2
votes
1answer
135 views

How to define \pgfkeyssetexpandedvalue?

Note: this question follows my previous question about \pgfkeyssetvalue. The \pgfkeyssetvalue is defined in pgfkeys.code.tex file by: \newtoks\pgfkeys@temptoks % [...] \long\def\pgfkeyssetvalue#1#2{...
3
votes
1answer
125 views

Why \pgfkeyssetvalue uses a token list?

The \pgfkeyssetvalue is defined by (in pgfkeys.code.tex file): \long\def\pgfkeyssetvalue#1#2{% \pgfkeys@temptoks{#2}\expandafter\edef\csname pgfk@#1\endcsname{\the\pgfkeys@temptoks}% } where \...
0
votes
1answer
317 views

Beamer boxes with variable sizes and position in frame

I want to to build flexible beamer boxes so that I can change the width and alignment within the frame. Passing the options width= and alignment=, whereas width is a valid value and alignment can be ...
5
votes
1answer
453 views

Obtaining the name of an undefined control sequence in LuaTeX

I want to implement something similar to \detokenize as a Lua function. But there is one case where I am prevented from accomplishing my goal. \directlua{tex.sprint("[" .. token.get_next().csname .. "...
12
votes
2answers
688 views

How are parameter tokens (#1,#2,…,#9) processed?

I wonder how the TeX token muncher processes parameter arguments. It seems that parameter arguments are sent token by token. Is this true? Consider the following code based on wipet's answer used to ...
4
votes
2answers
170 views

\tl_gput_right:Nn gets ! TeX capacity exceeded, sorry [input stack size=5000]

The following file gets ! TeX capacity exceeded, sorry [input stack size=5000]. If I remove the call to \tl_gput_right:Nn then it compiles successfully. \documentclass{article} \usepackage{amsfonts} \...
15
votes
1answer
165 views

Avoid @ hackery: How can I preserve spaces in a format specification when processing the specification with expl3 and PGF/TikZ?

This is a simplified version of part of the code I've been using in answers based on chronos, an attempt to create something somewhat like a substitute for the problematic chronosys. This code works ...
3
votes
2answers
90 views

Using token lists in a newenvironment definition

I'm trying to define a new environment to format some user scenarios for a project I'm working on. The MWE below shows what's happening. It appears that the first use of \\ empties the token lists. ...
4
votes
1answer
117 views

Is it possible to inject a token after another token?

Something like: If next token is some token that artificially creates a page break (e.g. \newpage, \clearpage Then inject something after it Else inject something before it I want to modify \input ...
5
votes
2answers
455 views

Why does \detokenize # sometimes fail inside of environments?

When I use the pound sign/number sign/hash sign # inside of commands and environments, this refers to arguments. However, shouldn't \detokenize{#} make it NOT an argument? Situation The pound sign ...
1
vote
0answers
59 views

Need help with grouping and expansion control

I am trying to define macros, which take a delimited list and pass each element to a macro (that takes one argument). I have fully working versions of them that iterate over 1) all elements in a list, ...
4
votes
2answers
104 views

How to process only tokens containing a lower case letter?

The goal is to have a macro with single argument, that looks at all tokens in turn and if it sees a lower-case letter it is wrapped in \texttt{<token>}. All other tokens are passed through ...
4
votes
1answer
95 views

Fake math sizing using tokens assignment

I'd like to implement sizing functions like \Huge in a way that makes them work in math mode as well. Essentially I want to create a command myHuge which maps {foo \myHuge bar} to {foo \mbox{\Huge $...
6
votes
1answer
207 views

LaTeX 3 token list varables, mapping functions, differences between mapping inline and functions

I am looking for a good explanation of the differences and use cases, between the tl_map_function:nN and tl_map_inline:Nn. I provide a MWE for the inline which also appears to be the fastest. \...
22
votes
5answers
2k views

Basics of parsing

I'm trying to master the art of parsing text one character at a time. The following is what I've discovered so far. Is this as good as it gets? I would really prefer not to do all those \edefs as ...
3
votes
3answers
353 views

How to randomly input a specified range of files?

The LaTeX file below outputs the numbers 1,2...,20, in natural order \documentclass[12pt]{article} \usepackage{multido} \begin{document} \multido{\i=1+1}{20}{\i\ } \end{document} The output is ...
9
votes
3answers
363 views

Appending active and expansion sensitive characters to a macro

This question comes up pretty often when people want to build tables with empty columns or similarly building up some active characters inside a macro hoping to expand them at the right time right ...
10
votes
5answers
452 views

Count number of entries with specific value

Assume I have a list with quite a few entries (all integer values, in case it's relevant), e.g., 1 2 4 0 3 1 0 0 2 1 7 1 4 2 3 0 0 6 5 2 1 2 3 0 Can LaTeX somehow count the number of 0's, 1's, ..., ...
5
votes
2answers
204 views

Difference between tokenlist and raw text in expl3

How to split token list by any char, like a raw document text? For example, i have a code to bold first word in macro argument. Macro is correctly works for direct using from code, like \bfirst{word ...
4
votes
1answer
164 views

Token replacement within a group

A comment below has clarified my problem, it has nothing to do with lua. In the code snippet below: \documentclass{article} \usepackage{xparse} \ExplSyntaxOn \cs_new_protected:Npn \...
6
votes
2answers
753 views

How to expand a LaTeX counter when adding to token list

In short: I am trying to append counter values to a token list, but when I print the token list at the end I get the final values of those counters. What I want is the value of the counters at the ...
2
votes
1answer
239 views

Token list with line breaks for writing multiple lines to file

I make use of a package (gnuplottex) that writes contents of an environment to a file, using the \verbatim@start/processline/end commands (by \immediate\write\FileHandle{\the\verbatim@line}). Now I ...
8
votes
1answer
327 views

Expandable, standard TeX only way to compare balanced token lists

Is there a (reasonably) efficient macro that does something similar to \long\def\comparets#1#2{\def\aa{#1}\def\bb{#2}\ifx\aa\bb true\else false\fi} except is expandable (i.e. \newcomparets{<...
3
votes
2answers
1k views

Extract first letter of each word in a sentence

I don't know if already exists a question like this, but after 20 minutes I haven't found one. There are similar ones, like How to define a command that bolds the first letter of each word in the ...
8
votes
2answers
267 views

Is it possible to set a particular item of a LaTeX3 seq?

I want to use a seq as an array and then set a particular item of this array by its index. I know that I can access an item via \seq_item:Nn but didn’t find out if there's something like \...
8
votes
2answers
239 views

How to count the frequency of a token in a token list

Is it possible to count the number of a token in a token list? \documentclass{article} \usepackage{expl3} \ExplSyntaxOn \tl_new:N \tobi_list_tl \tl_set:Nn \tobi_list_tl {foo~bar~baz} \begin{...
8
votes
2answers
208 views

How do I get a token list with an end-of-line character in it?

The context of this question is that I have a command defined using xparse that slurps in a verbatim argument which can be long. I want to do a replacement on the resulting token list, but the token ...
3
votes
2answers
140 views

Replace and add a/to a specific sequence of tokens

Is there a way to find a specific sequence of tokens and add after it a command? Or is there a way to find a specific sequence of tokens and replace it with another one? And do it only if the sequence ...
1
vote
2answers
736 views

Spacing depending on optional subscripts and superscripts

I am totally new to LaTeX (use it for 2 days), so I am sorry if this has already been asked, but I spent the last 6 hours searching for a solution, but to no avail. (I stumbled upon a package xpeek ...
20
votes
1answer
3k views

Using \toks registers

I was reading this tutorial for TeX programming: http://pgfplots.sourceforge.net/TeX-programming-notes.pdf recommended in the 2-nd answer here: Where do I start LaTeX programming? It says: \toks&...
72
votes
4answers
5k views

What is a token?

Please forgive such a basic question, but I did some Google searching and looked around on this site, and while there are many threads and online resources that involve tokens, I can't seem to find a ...
4
votes
3answers
961 views

How are space tokens and empty lines processed by long commands (as their potential arguments)?

How are spaces and empty lines processed by long commands (i.e., those that do not accept paragraph breaks inside)? Are there different space tokens aside from " " and an empty line? It appears that ...
5
votes
2answers
506 views

Splitting delimited token list argument

I wrote a little piece of code (function \getfirst:w) that splits its delimited argument at the first embedded :. The code works well if used standalone, but I am unable to place it into the .code:n ...
9
votes
1answer
660 views

Using \typeout behaviour different with \newtoks vs. \def

I feel as I really should know this, but sadly I don't. Why is the output of \typeout{\the\myToken} not the same as having it printed in the PDF as \the\myToken. No problem with a \def, just with ...
6
votes
2answers
993 views

Tokenize and apply command

I have a string separated with commas. On that I want to get each element and apply a command. I have the following, which works. \documentclass[12pt]{article} \usepackage[trim]{tokenizer} \...
5
votes
2answers
216 views

Using Expl3 token-list variables where token lists are called for

The code below illustrates the trouble I’m having. It seems I cannot directly use a token-list variable where a token list is called for, but why, and how can I get the result I want? \documentclass{...
7
votes
1answer
661 views

Peek ahead for next token not in token-list

This question led to a new package: xpeek I’ve been developing my xpeek package with lots of help from you folks on TeX.SX. It’s intended to help write commands which, like the familiar \xspace, ...
5
votes
2answers
212 views

Macro for a code translator

Is there an existing macro that "translate" tokens using some code ? For example : \def\code{{{a,b},{b,c},{c,a}}} \translate[\code]{caac} % => would typeset "abba" or even better something like ...
6
votes
2answers
1k views

Passing multiple arguments from ProcessList (xparse) to a macro. Result to be used in tabular

I am making a macro that will take user input of a grading scheme and place it into a tabular environment. I would like the syntax to be as simple as possible and decided on: \tabulate{ eval1,%1 ; ...