Why doesn't a trailing tab typeset?

Question

Consider the following Plain TeX manuscript

\catcode9=12\relax% ASCII 9 is tab
.\ \ .\par%
.   
.%
\bye

It's impossible to tell, but there's a tab just after the dot on the third manuscript line. To make sure, here's the byte content of the file (each byte is represented by a two-digit hexadecimal number):

You can see the tab represented as the hex number 09 on the second line.

This manuscript typesets thus:

.  .
. .

Observe that there seem to be two spaces between the dots on the first line, but only one space between the dots on the second line.

However, I'd expect there two be two spaces between the dots on the second line too: one for the tab, and one for the carriage-return at the end of the line.

One way to explain this was suggested by egreg in this answer (to a different question), which says:

TeX reads one record at a time (a line in the input file, more or less) and discards the end-of-record terminator along with all spaces or tabs that immediately precede this end-of-record

This implies that the tab character is never seen by the TeX engine's "stomach", which explains the typeset outcome.

However this quote is not supported by the TeXbook (20th printing, Addison-Wesley 1991), as far as the tab character is concerned. Indeed, the TeXbook describes the same stage of the processing thus (p. 46):

TeX deletes any <space> characters (number 32) that occur at the right end of an input line.

Note the specification of the number 32, which is ASCII space.

The TeX source code for the input routine (viz. input_ln, defined in section 31, p. 16 of texdoc tex) is also quite particular about trailing spaces:

Trailing blanks are removed from the line; thus, either last=first (in which case the line was entirely blank) or buffer[last − 1]≠"␣".

So why is there only a single space between the dots on the second typeset line?

---

Incidentally, if the font is changed to cmtt, e.g.:

\font\myfont=cmtt14\myfont%
\catcode9=12%
.\ \ .\par%
.   
.%
\bye

the distances between the two dots on both lines are the same, however, as Steven B. Segletes' experiment shows, the tab is not seen by the engine's "stomach".

Answer 1

Some code findings for the "folklore" of egreg's answer.

TeX removes "blanks" at the end of an input line. This is done at a very early stage, just after reading the line before considering category codes and the input characters get tokenized.

Originally, these "blanks" are spaces only, but TeX distributions like TeX Live or MiKTeX extends them to include tabulators (horizontal tabulator). The snippets show the behavior for TeX, and pdfTeX. Not shown are XeTeX and LuaTeX that also remove spaces and tabulators at the end of input lines.

The code snippets come from TeX Live (2016).

• texk/web2c/tex.web:

  @ The |input_ln| function brings the next line of input from the specified
  file [...]
  Trailing blanks are removed from the line;
  [...]
  @p function input_ln(var f:alpha_file;@!bypass_eoln:boolean):boolean;
    {inputs the next line or returns |false|}
  var last_nonblank:0..buf_size; {|last| with trailing blanks removed}
  begin if bypass_eoln then if not eof(f) then get(f);
    {input the first character of the line into |f^|}
  last:=first; {cf.\ Matthew 19\thinspace:\thinspace30}
  if eof(f) then input_ln:=false
  else  begin last_nonblank:=first;
    while not eoln(f) do
      begin if last>=max_buf_stack then
        begin max_buf_stack:=last+1;
        if max_buf_stack=buf_size then
          @<Report overflow of the input buffer, and abort@>;
        end;
      buffer[last]:=xord[f^]; get(f); incr(last);
      if buffer[last-1]<>" " then last_nonblank:=last;
      end;
    last:=last_nonblank; input_ln:=true;
    end;
  end;
  

  The original TeX only removes spaces at the end of an input line. However, the Pascal version of input_ln will be overwritten by a more efficient C version, see the next code snippets.

• texk/web2c/tex.ch is a change file for tex.web:

  @x [3.31] l.933 - Do `input_ln' in C.
  @p function input_ln(var f:alpha_file;@!bypass_eoln:boolean):boolean;
  [...]
  end;
  @y
  We define |input_ln| in C, for efficiency. [...]
  @z
  

• texk/web2c/lib/texmfmp.c:

  /* Read a line of input as efficiently as possible while still looking
     like Pascal.  We set `last' to `first' and return `false' if we get
     to eof.  Otherwise, we return `true' and set last = first +
     length(line except trailing whitespace).  */
  
  #ifndef XeTeX /* for XeTeX, we have a replacement function in XeTeX_ext.c */
  boolean
  input_line (FILE *f)
  {
    [...]
  
    /* Trim trailing whitespace.  */
    while (last > first && ISBLANK (buffer[last - 1]))
      --last;
  
    [...]
  }
  

• texk/kpathsea/c-ctype.h:

  #ifndef isblank
  #define isblank(c) ((c) == ' ' || (c) == '\t')
  #endif
  
  #define ISBLANK(c) (isascii (c) && isblank ((unsigned char)c))
  

  isblank tests for space and tabulators, therefore both are removed at the end of an input line.

• texk/web2c/ChangeLog:

  Thu Oct 16 20:39:27 1997  Olaf Weber  <...>
  
      * `tex.ch`: [...]  Also, various changes
      for e-TeX (small rearrangements, introduces Init..Tini, remove
      tabs and trailing blanks).  From Peter Breitenlohner
      <...>.
  

  The change is very old, two decades ago in the last century.

Answer 2

The file I used is

\catcode9=12

.
.       %
.
\bye

Line 3 has .<tab>, line 4 has .<tab>%, line 5 has .<tab><space>

> hexdump tabs.tex 
0000000 5c 63 61 74 63 6f 64 65 39 3d 31 32 0a 0a 2e 09
0000010 0a 2e 09 25 0a 2e 09 20 0a 5c 62 79 65 0a      
000001e

This is what pdftex typesets:

Only the <tab> followed by % survives, because TeX Live implementations of TeX remove trailing spaces and tabs from lines, irrespective of their catcodes. I tried to find the reference, but apparently this is to be considered folklore.

The space between the first two periods is added by the end-of-line.

Update 2019

With the 2019 release of TeX Live, the TeX engines no longer remove traling tab tokens, but only spaces and the output you get is

Answer 3

REVISED ANSWER

I discovered that even though I was copy and pasting a TAB into TeXworks, the editor itself was doing a conversion to spaces in my ORIGINAL ANSWER. Thus, I used a different editor which I knew would preserve the TAB character in the file, and it shows that the keyboard TAB behaves like the ^^I "TeX TAB"...almost.

If the TAB is not at the line end, then the "keyboard-TAB" and "TeX-TAB" behave identically. If however, the TAB is at the end-of-line, the "keyboard TAB" is treated as a space, whereas the "TeX TAB" is still treated however redefined by TeX.

Conclusions:

1. Keyboard TABS and TeX-TABs (^^I) seem to be treated the same, except at the end of input lines.

2. Keyboard TABS are removed at the end-of-line (what David said), whereas ^^I TABS are not.

3. Use the TeX TAB ^^I to denote TABS in code, as editors are prone to do auto-conversion on your keyboard TABS otherwise.

The MWE (WARNING: copy/pasting this MWE into your editor may result in a conversion of the tabs into spaces):

With the TAB as defined by \TeX\par
% TWO EXPLICIT SPACES
x\ \ x\par%
% THE FOLLOWING PUTS AN EMPTY GROUP AFTER THE "KEYBOARD-TAB"; RESULT = 2 SPACES
x   {}
x\par%
% THE FOLLOWING TRAILS WITH A KEYBOARD TAB (WHAT THE OP TRIED); RESULT = 1 SPACE
x   
x\par
% THE FOLLOWING TRAILS WITH A "TeX-TAB"; RESULT = 1 SPACE
x^^I
x\par

With the TAB as catcode 12:\par
\catcode`\^^I=12 %
% TWO EXPLICIT SPACES
x\ \ x\par%
% THE FOLLOWING PUTS AN EMPTY GROUP AFTER THE "KEYBOARD-TAB"; RESULT = TAB GLYPH + SPACE
x   {}
x\par%
% THE FOLLOWING TRAILS WITH A KEYBOARD TAB (WHAT THE OP TRIED); RESULT = 1 SPACE
x   
x\par
% THE FOLLOWING TRAILS WITH A "TeX-TAB"; RESULT = TAB GLYPH PLUS SPACE
x^^I
x\par

\catcode`\^^I=\active %
\def^^I{\space}
With the TAB as an active space

% TWO EXPLICIT SPACES
x\ \ x\par%
% THE FOLLOWING PUTS AN EMPTY GROUP AFTER THE "KEYBOARD-TAB"; RESULT = 2 SPACES
x   {}
x\par%
% THE FOLLOWING TRAILS WITH A KEYBOARD TAB (WHAT THE OP TRIED); RESULT = 1 SPACE
x   
x\par
% THE FOLLOWING TRAILS WITH A "TeX-TAB"; RESULT = 2 SPACES
x^^I
x\par

\catcode`\^^I=\active %
\def^^I{Q}
With the TAB as an active Q

% TWO EXPLICIT SPACES
x\ \ x\par%
% THE FOLLOWING PUTS AN EMPTY GROUP AFTER THE "KEYBOARD-TAB"; RESULT = Q +  SPACE
x   {}
x\par%
% THE FOLLOWING TRAILS WITH A KEYBOARD TAB (WHAT THE OP TRIED); RESULT = 1 SPACE
x   
x\par
% THE FOLLOWING TRAILS WITH A "TeX-TAB"; RESULT = Q + SPACE
x^^I
x\par

\bye

ORIGINAL ANSWER (AS FOOLED BY MY EDITOR)

The original answer was removed, because I was fooled by my editor when pasting a TAB into the input file...the editor did an auto-conversion to spaces.

---

I should note the TeXbook talks about the TAB character, as noted in my comments to the OP, on pages 8, 45, 369-370, and 391.

Answer 4

If you typeset a character 9 using the default plain tex font encoding you get a capital greek letter (all the Greek uppercase letters are in the ascii control code slots in cm fonts)

\catcode9=12\relax% ASCII 9 is tab


[   ]% tab (which will be converted to space by this site)

[ ]



\bye

the tabs are removed from ends of lines before tokenisation actually the default input_ln function in tex.web only strips spaces

    if buffer[last-1]<>" " then last_nonblank:=last;

But file handling is one of the few system-dependent tex features, and web2c based TeX also strip tab.

Answer 5

The behavior has been confirmed as a bug by David Fuchs, and TeX Live's Karl Berry has told me it would be fixed in web2c. Julian Gilbey traced the bug to the same source as Heiko Oberdiek did in his answer.

Here's what Mr. Fuchs had to say about this bug.

[...] yes this does seem to be porting bug; the correct output consists of two lines: "dot space space dot", "dot PSI space dot". (The upper-case PSI character is in position 9 of cmr10, and the tab character is 9 in ascii, of course, and the first line of the input file sets the catcode to make tabs be typeset.)

A slight subtlety is that TeX as specified exactly in Volume B does not accept any characters below <space>, including <tab>, at all; so pure? / minimalist? / fundamental? / nominal? TeX on this input simply grouses about the tab right away:

! Text line contains an invalid character.l.3 .^^?

But module 23 is very explicit on the proper way to "get the most permissive character set", which every port typically does, and it's in that context that the claim of "correct behavior" above is made.

Finally, one might wonder why TeX strips trailing spaces and nothing else, including tabs. The answer is that originally it didn't strip anything. But there were systems we wanted to support (particularly IBM's OS360 and VM/CMS) that had fixed-record conventions for text files. So, typically, moving a .tex file to an IBM mainframe caused each line to be padded out to 80 characters by the system adding trailing space characters. We wanted to make sure that any round-trip to/from such systems would not create an input file that had any chance of a mysterious change of behavior between platforms (in, say, some verbatim mode), and the best solution was to have TeX always ignore trailing space characters, so that it would be immune to them automagically appearing. But there was no reason to strip any other characters, including tab.

Answer 6

There are multiple matters here that have been brought up in various iterations of the question, comments, and answers:

1. what TeX does with trailing tabs at the end of a line
2. after reading in lines, how TeX treats tabs
3. how the visual width of a space is affected by the font
4. how TeX treats punctuation, specifically the .

I know your ultimate question is only about the first one, so let's get the other things out of the way first.

Consider the following test file, which is intended to illustrate the same points as your example:

\def\test{
a b

a\ b

a\ \ b

% There's a tab in the line below
a   b

% There's a tab at the end of the line below
a   
b
}

\def\testdot{
. .

.\ .

.\ \ .

% There's a tab in the line below
.   .

% There's a tab at the end of the line below
.   
.
}

\test

\testdot

\vskip 1em \hrule \vskip 1em

\font\myfont=cmtt14\myfont

\test

\testdot

\end

This produces:

Note that the .s behave differently from the a-b example except in the cases with explicit spaces (.\ . and .\ \ .). However, these differences are just because TeX typesets a larger space after sentences for typographic reasons (and all these differences go away if you add \frenchspacing), so we might as well work with just simple letters to avoid that confusion.

So with that out of the way, here is a simpler case:

\catcode9=12

% There's a tab in the line below
a   b

% There's a tab at the end of the line below
a   
b

\end

produces:

Conclusion: Trailing tabs are removed when reading a line.

---

Now your remaining question is why this is so, and whether this is consistent with Knuth's intentions as stated in either The TeXbook or the TeX program (aka Volume A and Volume B).

In section 31 of the TeX program is the input_ln procedure which reads in lines. He says “trailing blanks are removed”.

He also says this is something TeX implementations are encouraged to rewrite and optimize:

His own implementation is something that strips any trailing character whose xord is 32 (the fact that " " which appears as below actually means 32 is an aspect of WEB and string pool files…):

Note the xord there: it converts any character to an integer. And in the "official" implementation as in the book, the question of tabs in the input file has a simpler answer: they are treated as invalid:

Strictly speaking this means that if you had a hypothetical TeX implementation straight out of tex.web and with absolutely no system-dependent changes so that it used tex.web's implementation of input_ln and the same xord array (even Knuth never used such an implementation), then you couldn't even have a tab character in your file, anywhere: it would have an xord of invalid_code = 127, and when TeX encountered it, it would throw an error. (You can still see such an error by introducing byte 127 in a file: the error message is ! Text line contains an invalid character.)

As Heiko's answer points out, the web2c implementation of TeX as used in TeX Live implements input_ln in C for speed, as suggested by Knuth in section 31. It strips both trailing spaces and trailing tabs. This is consistent with a certain interpretation of "trailing blanks are removed" (which is probably why the change was considered ok). Note that by itself, it's not automatically inconsistent with the TeXbook on p. 46 saying that “TeX deletes any ⟨space⟩ characters (number 32) that occur at the right end of an input line” — that only refers to what happens after the file's bytes are translated using xord. This is what Knuth means when on pp. 44–45 of The TeXbook it says:

the people who installed your local TeX system can tell you the correspondence between what you type and the character number that TeX receives

So deleting these trailing tabs would have been perfectly consistent with the tex.web code of input_ln (and what's described in The TeXbook above), if its xord had been set up so that tabs are always treated as spaces (this is not the case). Instead, it follows a setting of xord and xchr where it appears all bytes are allowed in files and (other than a newline) byte N is read by TeX as N (nothing else gets translated to invalid_code). For curiosity, I created a file containing a<character C><newline>b for every non-printable character C from 0 to 127 (that is: 0 to 31, and 127). I also added a few \catcode instructions so that all of these (except for 13) have catcode 12. This is the result:

% xxd -g 1 -c 24 mwe.tex
00000000: 5c 63 61 74 63 6f 64 65 30 3d 31 32 0a 5c 63 61 74 63 6f 64 65 31 3d 31  \catcode0=12.\catcode1=1
00000018: 32 0a 5c 63 61 74 63 6f 64 65 39 3d 31 32 0a 5c 63 61 74 63 6f 64 65 31  2.\catcode9=12.\catcode1
00000030: 31 3d 31 32 0a 5c 63 61 74 63 6f 64 65 31 32 3d 31 32 0a 5c 63 61 74 63  1=12.\catcode12=12.\catc
00000048: 6f 64 65 31 34 3d 31 32 0a 5c 63 61 74 63 6f 64 65 31 32 37 3d 31 32 0a  ode14=12.\catcode127=12.
00000060: 61 00 0a 62 0a 0a 61 01 0a 62 0a 0a 61 02 0a 62 0a 0a 61 03 0a 62 0a 0a  a..b..a..b..a..b..a..b..
00000078: 61 04 0a 62 0a 0a 61 05 0a 62 0a 0a 61 06 0a 62 0a 0a 61 07 0a 62 0a 0a  a..b..a..b..a..b..a..b..
00000090: 61 08 0a 62 0a 0a 61 09 0a 62 0a 0a 61 0b 0a 62 0a 0a 61 0c 0a 62 0a 0a  a..b..a..b..a..b..a..b..
000000a8: 61 0d 0a 62 0a 0a 61 0e 0a 62 0a 0a 61 0f 0a 62 0a 0a 61 10 0a 62 0a 0a  a..b..a..b..a..b..a..b..
000000c0: 61 11 0a 62 0a 0a 61 12 0a 62 0a 0a 61 13 0a 62 0a 0a 61 14 0a 62 0a 0a  a..b..a..b..a..b..a..b..
000000d8: 61 15 0a 62 0a 0a 61 16 0a 62 0a 0a 61 17 0a 62 0a 0a 61 18 0a 62 0a 0a  a..b..a..b..a..b..a..b..
000000f0: 61 19 0a 62 0a 0a 61 1a 0a 62 0a 0a 61 1b 0a 62 0a 0a 61 1c 0a 62 0a 0a  a..b..a..b..a..b..a..b..
00000108: 61 1d 0a 62 0a 0a 61 1e 0a 62 0a 0a 61 1f 0a 62 0a 0a 61 7f 0a 62 0a 0a  a..b..a..b..a..b..a..b..
00000120: 5c 65 6e 64 0a                                                           \end.

Summary:

• In the web2c implementation of C, the xord of having byte c in the input file results in TeX receiving input code c. (This is allowed by Knuth's conventions: appendix C of The TeXbook is entirely about this kind of thing.)
• In the web2c implementation of C, trailing spaces and tabs are removed. This is inconsistent with what is described in The TeXbook, though it would have been consistent if the xord of a tab character were 32. (Though in that case plain.tex's definition of \catcode`\^^I=10 would have been mostly unnecessary (applying only when someone explicitly wrote ^^I in their input file), as a tab character in the input file would already be read by input_ln as a space.)