# Character bytes and character tokens: If newlines are converted to spaces, then where does catcode 5 come into the picture?

My goal is to understand the relationship between character bytes and character tokens with respect to new line bytes. I likely do not have my facts straight.

When TeX reads a file of bytes, encoding must be considered. Putting that aside,

• We can observe that a single new line character byte (assuming LF and CRLF) is converted into a space. But what happens behind the scenes? Is a token created using data pair (LF byte number, catcode=10)?

• Two consecutive new line character bytes become one single token with the data pair (space byte number, catcode 5)?

# When does catcode 5 "end of the line" come into play?

I know LaTeX inserts a \par when two consecutive line endings are encountered.

# Code

I attempted to visually show tokens with catcode 5, but I am still not sure if \tmp truly becomes catcode 5.

\documentclass{article}
\usepackage{fontspec}% xelatex

\long\def\scan#1{#1\par\rule{\textwidth}{2pt}\par\xscan#1\relax}
\long\def\xscan{\afterassignment\xxscan\let\tmp= }
\long\def\xxscan{%
\ifx\tmp\relax\else%
\ifcat\tmp\space10 \else%
\ifcat\tmp a11 \else%
\ifcat\tmp 112 \else%...
\ifcat\tmp

5
\else%
\fi\fi\fi\fi
\expandafter\xscan
\fi}

\begin{document}
\scan{  mac::exception  ==

a
}
\end{document}


# Notes

There are never any tokens with catcode 5.

Initex sets up

\catcode\^^M=5


But that acts in a similar way to

\catcode\%=14


which makes % catcode 14, but there are no tokens with that catcode, If a character with catcode 14 is scanned then that character and the rest of the line are discarded.

A character of catcode 5 generates a space character and puts tex's scanner in a special mode that causes immediately following characters of catcode 10 to be discarded "white space at the beginning of a line" and a following character of catcode 5 to be tokenised as \par not a space token "blank line is equivalent to \par"

So note that the first newline always makes a space, subsequent ones make \par so a blank line is usually equivalent to space\par.

• Aha! So I guess my 5 should be a 0 representing the \ in \par. Come to think of it, I do not know what a macro token data pair looks like. My weak guess: (char 9, catcode 0). I just bought "TeX in Practice - Volume III" to help my understanding here. – Jonathan Komar Jan 20 '17 at 10:04
• no there are no tokens of catcode 0 either, \ifcat just treats any non-expandable, non character token such as \par, or \relax or \halign as equal (which you may think of as a nominal catcode 16, although you can never actually see a numeric value) – David Carlisle Jan 20 '17 at 10:07
• I'd say that category code 5 generates a space character, not a space token. – egreg Jan 20 '17 at 10:44
• @egreg so do I:-) – David Carlisle Jan 20 '17 at 10:51

Characters have a category code; they can generate a character token during the tokenization phase, but they need not to.

Category codes are used with twofold purposes: they are looked at during tokenization (when TeX absorbs text from an input file or the terminal) but also during token list processing.

Only characters with category code

1 2 3 4 6 7 8 10 11 12 13

can generate character tokens (with the same category code), respectively

begin group
end group
math shift
alignment
parameter
superscript
subscript
space
letter
other character
active character

Characters with category code 0 5 9 14 15 will never generate a character token with the same category code: there is no way a character token with those category codes can get through in TeX internal token processor:

• a character with category code 0 triggers the formation of a control sequence

• a character with category code 9 is ignored

• a character with category code 15 raises an error and then it is ignored

• a character with category code 14 tells the tokenization processor to ignore it together with all other characters on the line

More interesting is category code 5, which is the object of your question. When TeX finds one, it discards whatever remains on the input line, generates a space character with character code 32 and category code 10 as if it had been on the line to begin with and sets the scanner in the special state of ignoring blank spaces (category code 10) until coming to something different: if this is another character of category code 5, TeX generates a \par token, otherwise enters the normal state.

Note the emphasis on space character above: this space character is tokenized according to the normal rules, so it will get ignored if it follows a control word (like \foo) but not after a control symbol (like \~).

A consequence of this is that the following inputs

\foo\baz

\foo \baz

\foo \baz

are completely equivalent. Note that if the end-of-line in the last input generated a space token, there would be a difference. But indeed a space character (not yet tokenized) is generated instead.

Note. What said above about ignored characters might be misleading when confronted with control word formation. The formation of a control word starts with a category code 0 character followed by one of category code 11. Any character with category code different from 11 will stop the scanning, cause tokenization of the formed control word and be examined anew (for being ignored, for instance, in case it has category code 9).

Addendum about XeTeX and LuaTeX. When a UTF-8 encoded file is fed to the Unicode aware engines, it's immaterial whether a character is single, two, three or four byte long in its UTF-8 representation. These two engines do a preliminary step transforming UTF-8 combinations into Unicode entities, so what the tokenization processor sees is just one character (with its category code as assigned in the initialization table). The two engines are also able to cope with UTF-16 or UTF-32, little or big endian.

TeX does read input line by line: A line of input will be read and processed. Then another line of input will be read and and processed. ...

One of the first things that TeX does after reading a line of input is deleting any characters of character code 32(space character) at the right end of the line.

Then TeX inserts a character at the right end of the line whose character code equals the value of the integer-parameter \endlinechar.
Usually the value of \endlinechar is 13 and the category code of character 13(return character) is 5(end of line).

This implies that usually TeX encounters a character whose category code is 5(end of line) when during tokenizing the line reaching the end of the line.

Then TeX starts tokenizing the line. I.e., TeX "looks" at the characters which the line contains and produces control sequence tokens and character tokens according to the category code table and according to the state of the reading apparatus.

At the time of reading and tokenizing input, the reading apparatus of TeX can be in one of three states:

State S: Skipping blanks. The reading apparatus will be in state S

• after processing a character from the input whose category code is 10(space).
• after processing a sequence of two equal characters of category code 7(superscript) followed by a sequence of two characters forming the character code in lowercase hexadecimal notation of a character whose category code is 10(space).
[Example: Usually the category code of ^ is 7(superscript) while the category code of character 32(space character; hex 20) usually is 10(space). Therefore the notation ^^20 usually is treated like processing the character 32(space character) from the input whose category code usually is 10(space).]
• after processing a sequence of two equal characters of category code 7(superscript) followed by a character where - in case of the character's character code being in the range from 64 to 127 - the category code of the character whose character code is obtained by subtracting 64 is 10(space).
[Example: As the category code of ^usually is 7(superscript) and the character code of  is 96 while 96-64=32 and the category code of character 32(space character) usually is 10(space), the notation ^^ usually is treated like processing the character 32(space character) from the input whose category code usually is 10(space).]
• after processing a sequence of two equal characters of category code 7(superscript) followed by a character where - in case of the character's character code being in the range from 0 to 63 - the category code of the character whose character code is obtained by adding 64 is 10(space).
• after producing a control word token.
• after producing the control symbol token \␣ (control space).

While in state S, both processing a character whose category code is 10(space) and processing a ^^..-sequence/<superscript-char><superscript-char>..-sequence considered equivalent to a character of category code 10(space) will yield not producing any token and not changing the state of the reading apparatus.

Usually the space character (character code 32) is the only character whose category code is 10(space).

That's why cou can have several consecutive space characters in the input usually yielding only one space token, in turn yielding whatsoever horizontal glue for only one horizontal space in case of TeX being in one of the modes where space tokens yield horizontal glue (, i.e., in horizontal mode, in restricted horizontal mode but neither in vertical mode, nor in internal vertical mode, nor in math mode, nor in display math mode).

State M: Middle of line. The reading apparatus will be in state M

• after producing a non-space character token.
• after producing a control symbol token other than the control symbol token \␣ (control space).

While in state M, both processing a character whose category code is 10(space) and processing a ^^..-sequence/<superscript-char><superscript-char>..-sequence considered equivalent to a character of category code 10(space) will yield producing a space token, i.e., a character token whose charcode is 32(space character) and whose catcode is 10(space), and switching the state of the reading apparatus to state S.

State N: New line. The reading apparatus is in state N when about to start reading another line of input. While in state N, both processing a character whose category code is 10(space) and processing a ^^..-sequence/<superscript-char><superscript-char>..-sequence considered equivalent to a character of category code 10(space) will yield not producing any token and not changing the state of the reading apparatus.

If TeX encounters a character of category code 5(end of line) while the reading apparatus is in state S, TeX will not produce any token at all.

If TeX encounters a character of category code 5(end of line) while the reading apparatus is in state M, TeX will produce a space token, i.e., a character token whose characode is 32(space character) and whose catcode is 10(space).

If TeX encounters a character of category code 5(end of line) while the reading apparatus is in state N, TeX will produce the control word token \par.

After encountering a character of category code 5(end of line), TeX will - no matter what state the reading apparatus is in - in any case drop any further information on the current line and start reading another line of input. Hereby the reading apparatus of TeX will be switched to state N.

Due to the above-mentioned \endlinechar-thingie usually an empty line after a non-empty line will yield having TeX process two consecutive return-characters (character code 13) whose category code is 5(end of line).

At the time of encountering the first of these return-characters, which is in the non-empty line, the reading apparatus might be in state S or in state M and therefore the first one might yield no token at all or yield a space token.
In any case after encountering the first of these return-characters, the reading apparatus will be switched to state N. Therefore at the time of encountering the second of these return-characters, the reading apparatus will be in state N and the second of these return-characters will yield the control word token \par.
That's why usually an empty line is treated like a "paragraph break"/like the control word token \par which usually (if not redefined) is a directive for breaking into lines and typesetting as a paragraph of text the material collected/gathered by now.

This means that it can happen that the last thing within a paragraph is a space token yielding horizontal glue. Be aware that such horizontal glue at the end of a paragraph usually gets discarded by TeX and glue according to the values of the \parfillskip-glue-parameter gets attached at the end of a paragraph.

• Can't you simplify your "State S" down to 2 ORs in that you remove space and just say "catcode 10 token or control sequence token"? or something to that effect... Afterall, I could have deviously set space to category 8... – Jonathan Komar Jan 23 '17 at 17:17
• @macmadness86 State S, State M and State N of the reading-apparatus are explained in the TeXbook using that terminology. Processing a character in the input whose category code is 10 is not the same as producing a space token: Processing a character whose category code is 10 while reading apparatus is in state S will lead to not producing any token at all while reading apparatus remains in state S. Usually _(underscore) is of category code 8. If you set space's (character 32's) category code to 8, you can use space only in the same ways as the underscore-character can be used. – Ulrich Diez Jan 23 '17 at 18:08
• @macmadness86 Be aware that characters are not the same as character-tokens. When you use an editing-software for typeing a .tex-input-file, that file will contain characters encoded by bytes. When TeX's reading-and tokenizing-apparatus comes into action, it will - line by line - process these characters according to rules given by the current catcode-table and the by the current state (M or N or S) of the reading-apparatus. Processing the characters may yield insertion of whatsoever tokens into the "token-stream". It may also yield other things, e.g. changing reading-apparatus' state or – Ulrich Diez Jan 23 '17 at 18:15
• @macmadness86 It may also yield other things, e.g. changing reading-apparatus' state or - in case of the comment-character % - dropping the remainder of the current line and starting processing the next line of input. – Ulrich Diez Jan 23 '17 at 18:17
• I have reread your answer. I would suggest a slight demonstrative word choice change: every place where you used "that state" only made sense when I understood "this state,". The difference would indicate that you are referring to the TeX state at hand rather than some other state. If you'd like, I could make the necessary adjustments. I'll just wait awhile for now. Also, the order in which you present the states seems out of practical order. Wouldn't the initial state be State N and therefore should come first in your explanation? – Jonathan Komar Jan 26 '17 at 16:27