Questions tagged [tokenization]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
3 votes
2 answers
255 views

What is the "code 2" when TeX tokenize?

On page 48 of The TeXbook, there is an exercise Assume that the category codes of plain TEX are in force, except that the characters ^^A, ^^B, ^^C, ^^M belong respectively to categories 0, 7, 10, and ...
Y. zeng's user avatar
  • 1,885
0 votes
1 answer
50 views

Language question: with \x defined as <X>, why using command \x is not the same as using <X>

This is intended more as a LaTeX language question than a "how do I accomplish XYZ" question. In the code below, a usage of VerbatimOut is successful. Place that usage inside a \newcommand* ...
David Fink's user avatar
0 votes
1 answer
155 views

ConTeXt: Limit number of words

Background Every chapter title contains multiple words, but only the first two words are to be used elsewhere (e.g., page header). Problem I'm looking for a general word splitting solution that can ...
Dave Jarvis's user avatar
  • 11.9k
7 votes
2 answers
461 views

How does \verb detect spaces that shouldn't exist

Consider the following MWE: \documentclass{article} \usepackage{listings} \lstset{basicstyle=\ttfamily} \begin{document} \lstinline |asdf|asdf asdfasdf \verb |asdf|asdf asdfasdf \end{document} ...
schtandard's user avatar
  • 14.9k
0 votes
1 answer
36 views

how to collect environment containing verbatim text and definitions with parameters to typset it later?

I'm stuck trying to collect tex tokens (detokenized) and output them latter with \scantokens. Problem is that can't drop duplicated parameters (#) Ideally would be great to read all the content as a ...
Linuxss's user avatar
  • 957
1 vote
0 answers
30 views

Eliding premature tokenisation in index macro

I am generating a very large number of index entries containing formatting in a LaTeX document, and some of them are in regions that affect the premature tokenisation of the argument (footnotes, ...
Peter Flynn's user avatar
  • 2,870