# Why doesn't a trailing tab typeset?

Consider the following Plain TeX manuscript

\catcode9=12\relax% ASCII 9 is tab
.\ \ .\par%
.
.%
\bye


It's impossible to tell, but there's a tab just after the dot on the third manuscript line. To make sure, here's the byte content of the file (each byte is represented by a two-digit hexadecimal number):

You can see the tab represented as the hex number 09 on the second line.

This manuscript typesets thus:

.  .
. .


Observe that there seem to be two spaces between the dots on the first line, but only one space between the dots on the second line.

However, I'd expect there two be two spaces between the dots on the second line too: one for the tab, and one for the carriage-return at the end of the line.

One way to explain this was suggested by egreg in this answer (to a different question), which says:

TeX reads one record at a time (a line in the input file, more or less) and discards the end-of-record terminator along with all spaces or tabs that immediately precede this end-of-record

This implies that the tab character is never seen by the TeX engine's "stomach", which explains the typeset outcome.

However this quote is not supported by the TeXbook (20th printing, Addison-Wesley 1991), as far as the tab character is concerned. Indeed, the TeXbook describes the same stage of the processing thus (p. 46):

TeX deletes any <space> characters (number 32) that occur at the right end of an input line.

Note the specification of the number 32, which is ASCII space.

The TeX source code for the input routine (viz. input_ln, defined in section 31, p. 16 of texdoc tex) is also quite particular about trailing spaces:

Trailing blanks are removed from the line; thus, either last=first (in which case the line was entirely blank) or buffer[last − 1]≠"␣".

So why is there only a single space between the dots on the second typeset line?

Incidentally, if the font is changed to cmtt, e.g.:

\font\myfont=cmtt14\myfont%
\catcode9=12%
.\ \ .\par%
.
.%
\bye


the distances between the two dots on both lines are the same, however, as Steven B. Segletes' experiment shows, the tab is not seen by the engine's "stomach".

• You mean a tab at the end of the third line, don't you? Aug 30, 2017 at 14:43
• Adding an empty group,{}, to the end of the 3rd line produces the extra space in the output, so clearly the tab and the end-of-line are being digested as a single space. Aug 30, 2017 at 14:46
• Page 45 TeXbook? "The control code ^^I is also of potential interest, since it’s the ASCII <tab>. Plain TEX makes <tab> act like a blank space." Aug 30, 2017 at 14:51
• @StevenB.Segletes: Your own experiment suggested in your first comment shows that it does not act like \relax. Aug 30, 2017 at 15:02
• BTW just noticed this duplicate(!) question: Which TeX procedure removes trailing tabs? Aug 30, 2017 at 23:54

Some code findings for the "folklore" of egreg's answer.

TeX removes "blanks" at the end of an input line. This is done at a very early stage, just after reading the line before considering category codes and the input characters get tokenized.

Originally, these "blanks" are spaces only, but TeX distributions like TeX Live or MiKTeX extends them to include tabulators (horizontal tabulator). The snippets show the behavior for TeX, and pdfTeX. Not shown are XeTeX and LuaTeX that also remove spaces and tabulators at the end of input lines.

The code snippets come from TeX Live (2016).

• texk/web2c/tex.web:

@ The |input_ln| function brings the next line of input from the specified
file [...]
Trailing blanks are removed from the line;
[...]
@p function input_ln(var f:alpha_file;@!bypass_eoln:boolean):boolean;
{inputs the next line or returns |false|}
var last_nonblank:0..buf_size; {|last| with trailing blanks removed}
begin if bypass_eoln then if not eof(f) then get(f);
{input the first character of the line into |f^|}
last:=first; {cf.\ Matthew 19\thinspace:\thinspace30}
if eof(f) then input_ln:=false
else  begin last_nonblank:=first;
while not eoln(f) do
begin if last>=max_buf_stack then
begin max_buf_stack:=last+1;
if max_buf_stack=buf_size then
@<Report overflow of the input buffer, and abort@>;
end;
buffer[last]:=xord[f^]; get(f); incr(last);
if buffer[last-1]<>" " then last_nonblank:=last;
end;
last:=last_nonblank; input_ln:=true;
end;
end;


The original TeX only removes spaces at the end of an input line. However, the Pascal version of input_ln will be overwritten by a more efficient C version, see the next code snippets.

• texk/web2c/tex.ch is a change file for tex.web:

@x [3.31] l.933 - Do input_ln' in C.
@p function input_ln(var f:alpha_file;@!bypass_eoln:boolean):boolean;
[...]
end;
@y
We define |input_ln| in C, for efficiency. [...]
@z

• texk/web2c/lib/texmfmp.c:

/* Read a line of input as efficiently as possible while still looking
like Pascal.  We set last' to first' and return false' if we get
to eof.  Otherwise, we return true' and set last = first +
length(line except trailing whitespace).  */

#ifndef XeTeX /* for XeTeX, we have a replacement function in XeTeX_ext.c */
boolean
input_line (FILE *f)
{
[...]

/* Trim trailing whitespace.  */
while (last > first && ISBLANK (buffer[last - 1]))
--last;

[...]
}

• texk/kpathsea/c-ctype.h:

#ifndef isblank
#define isblank(c) ((c) == ' ' || (c) == '\t')
#endif

#define ISBLANK(c) (isascii (c) && isblank ((unsigned char)c))


isblank tests for space and tabulators, therefore both are removed at the end of an input line.

• texk/web2c/ChangeLog:

Thu Oct 16 20:39:27 1997  Olaf Weber  <...>

* tex.ch: [...]  Also, various changes
for e-TeX (small rearrangements, introduces Init..Tini, remove
tabs and trailing blanks).  From Peter Breitenlohner
<...>.


The change is very old, two decades ago in the last century.

• Thank you. I've submitted a bug report to TeX Live. From your answer it looks like it will be a piece of cake to fix, if there's any desire to do so. Aug 30, 2017 at 21:29
• @EvanAad I do not know the reason, why tabulators are also removed at the end of an input line. But, since the main TeX distributions (TeX Live and MiKTeX) and its TeX compilers (TeX, pdfTeX, XeTeX, LuaTeX) behave the same way, I have some doubts that this will be "fixed". Aug 30, 2017 at 21:47
• There are also arguments for the removal of tabulators. Like the space, it is an invisible horizontal white space character. Most editors will not show them. LaTeX and plain TeX assign the tabulator the category code of a space. Then a tabulator at the line end with the line end might cause two spaces (a little difficult to test). This would be also a cause for a "bug report". Aug 30, 2017 at 21:51
• Good reasons, but do you think Knuth was unaware of them when he specified that only trailing spaces should be removed? I don't understand your last point. A tab followed by a carriage return will, under normal circumstances, result in only at most a single space, as per the usual rules for spaces. Aug 30, 2017 at 21:57
• @EvanAad I found a test by using a different character with category code 10, they get merged with the end of line (category code 5). Aug 30, 2017 at 22:01

The file I used is

\catcode9=12

.
.       %
.
\bye


Line 3 has .<tab>, line 4 has .<tab>%, line 5 has .<tab><space>

> hexdump tabs.tex
0000000 5c 63 61 74 63 6f 64 65 39 3d 31 32 0a 0a 2e 09
0000010 0a 2e 09 25 0a 2e 09 20 0a 5c 62 79 65 0a
000001e


This is what pdftex typesets:

Only the <tab> followed by % survives, because TeX Live implementations of TeX remove trailing spaces and tabs from lines, irrespective of their catcodes. I tried to find the reference, but apparently this is to be considered folklore.

The space between the first two periods is added by the end-of-line.

# Update 2019

With the 2019 release of TeX Live, the TeX engines no longer remove traling tab tokens, but only spaces and the output you get is

• Thanks. In my opinion, this constitutes a bug in the TeX Live implementation, considering that both the TeXbook and the TeX source code are particular about removing trailing spaces from the end of the line. Do you agree? Should I open a bug report? Aug 30, 2017 at 19:52
• @EvanAad This has been like this for several years. Aug 30, 2017 at 19:53
• @EvenAad: I agree it's technically a bug, but I suspect it's unlikely to get fixed. It would be interesting to have the original implementers chime in why they took this decision decades ago. Aug 30, 2017 at 19:59
• @BrunoLeFloch: I get the rationale. It's the same reason why <space> and <tab> are given the same catcode in Plain TeX. I mean, trailing spaces and trailing tabs are virtually indistinguishable. Nevertheless, it's safe to assume that this was not beyond Knuth's awareness when he wrote the code, and yet he chose to specify that only trailing spaces should be discarded. Shouldn't his specs be respected? Alternatively, maybe he can be convinced to change the specs. Aug 30, 2017 at 20:27
• @egreg Turns out this question has been asked and answered on this site before :-) tex.stackexchange.com/questions/66498/… Aug 31, 2017 at 0:03

I discovered that even though I was copy and pasting a TAB into TeXworks, the editor itself was doing a conversion to spaces in my ORIGINAL ANSWER. Thus, I used a different editor which I knew would preserve the TAB character in the file, and it shows that the keyboard TAB behaves like the ^^I "TeX TAB"...almost.

If the TAB is not at the line end, then the "keyboard-TAB" and "TeX-TAB" behave identically. If however, the TAB is at the end-of-line, the "keyboard TAB" is treated as a space, whereas the "TeX TAB" is still treated however redefined by TeX.

Conclusions:

1. Keyboard TABS and TeX-TABs (^^I) seem to be treated the same, except at the end of input lines.

2. Keyboard TABS are removed at the end-of-line (what David said), whereas ^^I TABS are not.

3. Use the TeX TAB ^^I to denote TABS in code, as editors are prone to do auto-conversion on your keyboard TABS otherwise.

The MWE (WARNING: copy/pasting this MWE into your editor may result in a conversion of the tabs into spaces):

With the TAB as defined by \TeX\par
% TWO EXPLICIT SPACES
x\ \ x\par%
% THE FOLLOWING PUTS AN EMPTY GROUP AFTER THE "KEYBOARD-TAB"; RESULT = 2 SPACES
x   {}
x\par%
% THE FOLLOWING TRAILS WITH A KEYBOARD TAB (WHAT THE OP TRIED); RESULT = 1 SPACE
x
x\par
% THE FOLLOWING TRAILS WITH A "TeX-TAB"; RESULT = 1 SPACE
x^^I
x\par

With the TAB as catcode 12:\par
\catcode\^^I=12 %
% TWO EXPLICIT SPACES
x\ \ x\par%
% THE FOLLOWING PUTS AN EMPTY GROUP AFTER THE "KEYBOARD-TAB"; RESULT = TAB GLYPH + SPACE
x   {}
x\par%
% THE FOLLOWING TRAILS WITH A KEYBOARD TAB (WHAT THE OP TRIED); RESULT = 1 SPACE
x
x\par
% THE FOLLOWING TRAILS WITH A "TeX-TAB"; RESULT = TAB GLYPH PLUS SPACE
x^^I
x\par

\catcode\^^I=\active %
\def^^I{\space}
With the TAB as an active space

% TWO EXPLICIT SPACES
x\ \ x\par%
% THE FOLLOWING PUTS AN EMPTY GROUP AFTER THE "KEYBOARD-TAB"; RESULT = 2 SPACES
x   {}
x\par%
% THE FOLLOWING TRAILS WITH A KEYBOARD TAB (WHAT THE OP TRIED); RESULT = 1 SPACE
x
x\par
% THE FOLLOWING TRAILS WITH A "TeX-TAB"; RESULT = 2 SPACES
x^^I
x\par

\catcode\^^I=\active %
\def^^I{Q}
With the TAB as an active Q

% TWO EXPLICIT SPACES
x\ \ x\par%
% THE FOLLOWING PUTS AN EMPTY GROUP AFTER THE "KEYBOARD-TAB"; RESULT = Q +  SPACE
x   {}
x\par%
% THE FOLLOWING TRAILS WITH A KEYBOARD TAB (WHAT THE OP TRIED); RESULT = 1 SPACE
x
x\par
% THE FOLLOWING TRAILS WITH A "TeX-TAB"; RESULT = Q + SPACE
x^^I
x\par

\bye


ORIGINAL ANSWER (AS FOOLED BY MY EDITOR)

The original answer was removed, because I was fooled by my editor when pasting a TAB into the input file...the editor did an auto-conversion to spaces.

I should note the TeXbook talks about the TAB character, as noted in my comments to the OP, on pages 8, 45, 369-370, and 391.

• Firstly I'd like to thank you for taking the time to read, ponder, and attempt to answer my question. However, I'm not going to accept your answer, and here's why. Regarding conclusion #1: my question was specifically about trailing tabs that are entered literally, not about any other kind of tabs. Regarding conclusion #2: this conclusion was the premise of my question; the question is: why does this conclusion hold, considering the TeXbook implicitly precludes it. Regarding conclusion #3: This is irrelevant to my question. Aug 30, 2017 at 18:27
• @EvanAad I understand completely. It was a much a learning exercise for me. To some extent, my answer is just a collection of what I learned, which, as you note, did not resolve your issue. I thank you for putting up with my ramblings. Aug 30, 2017 at 18:35
• @EvanAad Mildly related: tex.stackexchange.com/questions/311172/… Aug 30, 2017 at 19:23
• @StevenB.Segletes: Hi, maybe you can modify your example file whit real TABs using the tip given by Ulrich Diez at (tex.stackexchange.com/a/508103/7832) that is somehow related. Sep 13, 2019 at 3:00

If you typeset a character 9 using the default plain tex font encoding you get a capital greek letter (all the Greek uppercase letters are in the ascii control code slots in cm fonts)

\catcode9=12\relax% ASCII 9 is tab

[   ]% tab (which will be converted to space by this site)

[ ]

\bye


the tabs are removed from ends of lines before tokenisation actually the default input_ln function in tex.web only strips spaces

    if buffer[last-1]<>" " then last_nonblank:=last;


But file handling is one of the few system-dependent tex features, and web2c based TeX also strip tab.

• This is all well and good, but not what I asked. Your last sentence is the premise of my question. My question is: how can this be explained, considering the TeXbook implicitly states otherwise. Aug 30, 2017 at 17:57
• @EvanAad the TeXBook says: Another noteworthy characteristic of this manual is that it doesn't always tell the ^{truth}. You need to read tex.web not the texbook, but actually that seems to just be looking for space not space or tab, so I may adjust my answer later. Just looking... Aug 30, 2017 at 18:01
• Seems to me that @EvanAad is trying to find all the lies; I hope he gets all the jokes. Sep 5, 2017 at 16:46

The behavior has been confirmed as a bug by David Fuchs, and TeX Live's Karl Berry has told me it would be fixed in web2c. Julian Gilbey traced the bug to the same source as Heiko Oberdiek did in his answer.

[...] yes this does seem to be porting bug; the correct output consists of two lines: "dot space space dot", "dot PSI space dot". (The upper-case PSI character is in position 9 of cmr10, and the tab character is 9 in ascii, of course, and the first line of the input file sets the catcode to make tabs be typeset.)

A slight subtlety is that TeX as specified exactly in Volume B does not accept any characters below <space>, including <tab>, at all; so pure? / minimalist? / fundamental? / nominal? TeX on this input simply grouses about the tab right away:

! Text line contains an invalid character.l.3 .^^?

But module 23 is very explicit on the proper way to "get the most permissive character set", which every port typically does, and it's in that context that the claim of "correct behavior" above is made.

Finally, one might wonder why TeX strips trailing spaces and nothing else, including tabs. The answer is that originally it didn't strip anything. But there were systems we wanted to support (particularly IBM's OS360 and VM/CMS) that had fixed-record conventions for text files. So, typically, moving a .tex file to an IBM mainframe caused each line to be padded out to 80 characters by the system adding trailing space characters. We wanted to make sure that any round-trip to/from such systems would not create an input file that had any chance of a mysterious change of behavior between platforms (in, say, some verbatim mode), and the best solution was to have TeX always ignore trailing space characters, so that it would be immune to them automagically appearing. But there was no reason to strip any other characters, including tab.

• Nice. That's what i said too, about “nominal” TeX (as in Volume B) saying Text line contains an invalid character on tabs. How did you get in touch with these people? Sep 6, 2017 at 21:46
• @ShreevatsaR: Yes you did mention it. Well observed. All I did was post a bug report to the tex-k mailing list from the tug.org mailing lists. First Julian replied, then Karl, and Karl contacted David and let me know of his reply. Sep 6, 2017 at 22:38

There are multiple matters here that have been brought up in various iterations of the question, comments, and answers:

1. what TeX does with trailing tabs at the end of a line
2. after reading in lines, how TeX treats tabs
3. how the visual width of a space is affected by the font
4. how TeX treats punctuation, specifically the .

I know your ultimate question is only about the first one, so let's get the other things out of the way first.

Consider the following test file, which is intended to illustrate the same points as your example:

\def\test{
a b

a\ b

a\ \ b

% There's a tab in the line below
a   b

% There's a tab at the end of the line below
a
b
}

\def\testdot{
. .

.\ .

.\ \ .

% There's a tab in the line below
.   .

% There's a tab at the end of the line below
.
.
}

\test

\testdot

\vskip 1em \hrule \vskip 1em

\font\myfont=cmtt14\myfont

\test

\testdot

\end


This produces:

Note that the .s behave differently from the a-b example except in the cases with explicit spaces (.\ . and .\ \ .). However, these differences are just because TeX typesets a larger space after sentences for typographic reasons (and all these differences go away if you add \frenchspacing), so we might as well work with just simple letters to avoid that confusion.

So with that out of the way, here is a simpler case:

\catcode9=12

% There's a tab in the line below
a   b

% There's a tab at the end of the line below
a
b

\end


produces:

Conclusion: Trailing tabs are removed when reading a line.

Now your remaining question is why this is so, and whether this is consistent with Knuth's intentions as stated in either The TeXbook or the TeX program (aka Volume A and Volume B).

In section 31 of the TeX program is the input_ln procedure which reads in lines. He says “trailing blanks are removed”.

He also says this is something TeX implementations are encouraged to rewrite and optimize:

His own implementation is something that strips any trailing character whose xord is 32 (the fact that " " which appears as below actually means 32 is an aspect of WEB and string pool files…):

Note the xord there: it converts any character to an integer. And in the "official" implementation as in the book, the question of tabs in the input file has a simpler answer: they are treated as invalid:

Strictly speaking this means that if you had a hypothetical TeX implementation straight out of tex.web and with absolutely no system-dependent changes so that it used tex.web's implementation of input_ln and the same xord array (even Knuth never used such an implementation), then you couldn't even have a tab character in your file, anywhere: it would have an xord of invalid_code = 127, and when TeX encountered it, it would throw an error. (You can still see such an error by introducing byte 127 in a file: the error message is ! Text line contains an invalid character.)

As Heiko's answer points out, the web2c implementation of TeX as used in TeX Live implements input_ln in C for speed, as suggested by Knuth in section 31. It strips both trailing spaces and trailing tabs. This is consistent with a certain interpretation of "trailing blanks are removed" (which is probably why the change was considered ok). Note that by itself, it's not automatically inconsistent with the TeXbook on p. 46 saying that “TeX deletes any ⟨space⟩ characters (number 32) that occur at the right end of an input line” — that only refers to what happens after the file's bytes are translated using xord. This is what Knuth means when on pp. 44–45 of The TeXbook it says:

the people who installed your local TeX system can tell you the correspondence between what you type and the character number that TeX receives

So deleting these trailing tabs would have been perfectly consistent with the tex.web code of input_ln (and what's described in The TeXbook above), if its xord had been set up so that tabs are always treated as spaces (this is not the case). Instead, it follows a setting of xord and xchr where it appears all bytes are allowed in files and (other than a newline) byte N is read by TeX as N (nothing else gets translated to invalid_code). For curiosity, I created a file containing a<character C><newline>b for every non-printable character C from 0 to 127 (that is: 0 to 31, and 127). I also added a few \catcode instructions so that all of these (except for 13) have catcode 12. This is the result:

% xxd -g 1 -c 24 mwe.tex
00000000: 5c 63 61 74 63 6f 64 65 30 3d 31 32 0a 5c 63 61 74 63 6f 64 65 31 3d 31  \catcode0=12.\catcode1=1
00000018: 32 0a 5c 63 61 74 63 6f 64 65 39 3d 31 32 0a 5c 63 61 74 63 6f 64 65 31  2.\catcode9=12.\catcode1
00000030: 31 3d 31 32 0a 5c 63 61 74 63 6f 64 65 31 32 3d 31 32 0a 5c 63 61 74 63  1=12.\catcode12=12.\catc
00000048: 6f 64 65 31 34 3d 31 32 0a 5c 63 61 74 63 6f 64 65 31 32 37 3d 31 32 0a  ode14=12.\catcode127=12.
00000060: 61 00 0a 62 0a 0a 61 01 0a 62 0a 0a 61 02 0a 62 0a 0a 61 03 0a 62 0a 0a  a..b..a..b..a..b..a..b..
00000078: 61 04 0a 62 0a 0a 61 05 0a 62 0a 0a 61 06 0a 62 0a 0a 61 07 0a 62 0a 0a  a..b..a..b..a..b..a..b..
00000090: 61 08 0a 62 0a 0a 61 09 0a 62 0a 0a 61 0b 0a 62 0a 0a 61 0c 0a 62 0a 0a  a..b..a..b..a..b..a..b..
000000a8: 61 0d 0a 62 0a 0a 61 0e 0a 62 0a 0a 61 0f 0a 62 0a 0a 61 10 0a 62 0a 0a  a..b..a..b..a..b..a..b..
000000c0: 61 11 0a 62 0a 0a 61 12 0a 62 0a 0a 61 13 0a 62 0a 0a 61 14 0a 62 0a 0a  a..b..a..b..a..b..a..b..
000000d8: 61 15 0a 62 0a 0a 61 16 0a 62 0a 0a 61 17 0a 62 0a 0a 61 18 0a 62 0a 0a  a..b..a..b..a..b..a..b..
000000f0: 61 19 0a 62 0a 0a 61 1a 0a 62 0a 0a 61 1b 0a 62 0a 0a 61 1c 0a 62 0a 0a  a..b..a..b..a..b..a..b..
00000108: 61 1d 0a 62 0a 0a 61 1e 0a 62 0a 0a 61 1f 0a 62 0a 0a 61 7f 0a 62 0a 0a  a..b..a..b..a..b..a..b..
00000120: 5c 65 6e 64 0a                                                           \end.


Summary:

• In the web2c implementation of C, the xord of having byte c in the input file results in TeX receiving input code c. (This is allowed by Knuth's conventions: appendix C of The TeXbook is entirely about this kind of thing.)
• In the web2c implementation of C, trailing spaces and tabs are removed. This is inconsistent with what is described in The TeXbook, though it would have been consistent if the xord of a tab character were 32. (Though in that case plain.tex's definition of \catcode\^^I=10 would have been mostly unnecessary (applying only when someone explicitly wrote ^^I in their input file), as a tab character in the input file would already be read by input_ln as a space.)
• Thanks. The point about French spacing was enlightening. I also agree with your final two conclusions. Regarding the other points... Aug 31, 2017 at 6:45
• 1. You wrote: 'In section 31 of the TeX program is the input_ln procedure which reads in lines. He says “trailing blanks are removed”.' When Knuth writes "blanks", he means "␣", as he makes clear in no uncertain terms in the formula at the end of the very sentence from which the above quote was excerpted. As for what "␣" means - see 3 below. Aug 31, 2017 at 6:45
• 2. You wrote: "He also says this is something TeX implementations are encouraged to rewrite and optimize". In the quote you cited he clarifies exactly what optimization is allowed ("special routines that read in an entire array of characters at once") and what its purpose is ("to reduce system overhead"). Removing trailing tabs is not an "optimization" that falls under these criteria. Aug 31, 2017 at 6:45
• 3. You wrote: 'His own implementation is something that strips any trailing character whose xord is 32 (the fact that " " which appears as "␣" below actually means 32 is an aspect of WEB and string pool files…):' I disagree with your interpretation. "␣" means "the space character, in whatever encoding the Pascal system used to compile the source code is set up to use". See section 21 on p. 11. If Knuth wanted to write buffer[last - 1] ≠ xord['40], he would have done so. Aug 31, 2017 at 6:46
• @EvanAad Thanks. I mostly agree with your comments; I only have a comment about your (3): when you run tangle on tex.web, it turns the line if buffer[last - 1] ≠ "␣" into if buffer[last-1]<>32 — the fact that " " means 32 (on all systems) is a feature of WEB. So the double-quoted string " " doesn't really mean “the space character, in whatever encoding the Pascal system used to compile the source code is set up to use”, although the single-quoted ' ' means exactly what you said. (But again, this 32 is after xord` has converted the file's character to an integer…) Aug 31, 2017 at 7:24