Why do spaces stop my hrules from all being flush with each other?

Question

I'm trying to make a typeset form using some custom commands that I've built. Part of that effort is making "blanks": macros for blank space where someone can write something in after the document has been printed. Their definition is in the formfields package (information about how to get that is at the bottom of the post).

I ran into the following issue: I had some of the blanks inside of tables, and the blanks in the left column weren't coming up flush to the end of first column. Eventually, I traced that to the following change, demonstrated in these two tables from this MWE:

\documentclass[draft]{article}
\usepackage{tabu}
\usepackage{formfields}
\usepackage[left=1.0in, right=.5in, top=.7in, bottom=.7in]{geometry}

\begin{document}
% not flush, and blanks don't all come up to same place.
\begin{tabu} {X<{ \hspace{.5em} }>{ \hspace{.5em} }X}
    \field{thing}& \hrulefill\\
    \blank & \hrulefill\\
    \field{thing}& \hrulefill\\
\end{tabu}

% all blanks flush with each other at right end of 1st column.
\begin{tabu} {X<{\hspace{.5em}}>{\hspace{.5em}}X}
    \field{thing}& \hrulefill\\
    \blank & \hrulefill\\
    \field{thing}& \hrulefill\\
\end{tabu}

% same as second table.
\begin{tabu} {XX}
    \field{thing}& \hrulefill\\
    \blank & \hrulefill\\
    \field{thing}& \hrulefill\\
\end{tabu}
\end{document}

Why do the blanks and fields all come up flush against the end of the first column in the second and third tables, but not the first? I could understand if I were doing something different to each row, but in both of these tables, I don't think I'm doing anything different that would make the rules not align with each other on the right side; I assume this means the spaces at the end of the first column in the different rows of the 1st table are of different lengths. I don't see why that should be the case. I ask this question because I couldn't figure out that the spaces in my table header were the culprit for several weeks. I'm hoping to get some insight as to how to diagnose and debug this sort of thing. If you can, please pay special attention to that.

Part of the problem seems to be the spaces within the < and > column specifiers in my table, because the second table from the MWE has the spaces removed, and all the blanks come up flush with the end of the first column. I'm not suprised the spaces stop the blanks from becoming flush with the right end of the 1st column, but I don't understand why the different blanks would act differently.

Additional Resources

A picture of the trouble (the red rectangles were added later, they show where the lines aren't flush with each other).

Zoomed in picture of the first (top) table:

formfields.sty can be obtained here

Answer 1

One way to debug this is to look at the produced output box. You can see a representation of it when you add

\tracingoutput=1
\showboxbreadth=100
\showboxdepth=20

at the begin of your document. You can then compare the differences between the three tables and its cells so that you at least see to what result TeX has come.

Why is there extra space only in some places?

This is the part further investigation is necessary. In table 3 it's pretty obvious, there's no space defined in the column formats, so no space is added to the output. In table 2 only \hspaces, internally \hskips, i.e. glue elements, are added to the output boxes. In table 1 some of the space characters also find their way into the output, perhaps due to the way TeX parses macro parameters.

Whenever a glue appears at the begin and end of a box/line, it is removed from the output. This is why the \hspace is removed from table 2/column 1. However, in table 2/column 2 it's still there at the left because the \null box (which is there in column 1 too) appears before it. I have no idea why the \hspace is still gone in column 1.

Why are there difference spaces in the first table?

Let's compare the relevant parts of the boxes of lines 1 and 2 of table 1/column 1:

% line 1                                        line 2
  \hbox(0.0+0.0)x0.0                            \hbox(0.0+0.0)x0.0
  \OT1/cmr/m/n/10 t
  \OT1/cmr/m/n/10 h
  \OT1/cmr/m/n/10 i
  \OT1/cmr/m/n/10 n
  \OT1/cmr/m/n/10 g
  \OT1/cmr/m/n/10 :
  \leaders 3.50006
    \rule(*+*)x0.4

  \leaders 0.0 plus 1.0fill                     \leaders 0.0 plus 1.0fill
    \rule(0.4+0.0)x*                              \rule(0.4+0.0)x*
  \kern 0.0                                     \kern 0.0
  \glue 4.44444 plus 3.33331 minus 0.55556      \glue 3.33333 plus 1.66666 minus 1.11111
%       ^------------ here's the difference ----------^
  \glue 5.0                                     \glue 5.0

  \penalty 10000                                \penalty 10000
  \rule(0.0+0.0)x0.0                            \rule(0.0+0.0)x0.0
  \penalty 10000                                \penalty 10000
  \glue(\parfillskip) 0.0 plus 1.0fil           \glue(\parfillskip) 0.0 plus 1.0fil
  \glue(\rightskip) 0.0                         \glue(\rightskip) 0.0

You can see the two \glue values here in the middle part. The first is from one of the space characters, the second from the \hspace. Both should get their natural width, as there's the leader with a 1.0fill that occupies all the remaining space. Now why is the first one larger than the second?

The answer is the way TeX handles spaces after punctuation marks. By \sfcode, each character in TeX can be assigned a space factor to which can stretch the space after it. Because your \field macro prints a : at the end, the active space factor when the space character is processed is still the one assigned to :, which is 2000 in the default setting. This makes the following space slightly larger than the one at natural width in line 2 by adding a so called "extra space".

Try adding \sfcode`\:=1000 at the begin of your document to force the natural width and the spaces at the end become the same width:

Answer 2

It's not at all clear why you're adding \hspace in the column specifications.

Anyway,

\begin{tabu} {X<{ \hspace{.5em} }>{ \hspace{.5em} }X}

does much more than adding .5em, because it adds “space”, “half a quad”, “space”. However, in the first column the final space is removed when the paragraph is finished up. This also happens in your second example, where the trailing \hspace{.5em} is eaten up.

You could use {\hspace*{.5em}}, but there's a much simpler solution.

\documentclass[draft]{article}
\usepackage{tabu}
\usepackage{formfields}
\usepackage[left=1.0in, right=.5in, top=.7in, bottom=.7in]{geometry}

\begin{document}
% not flush, and blanks don't all come up to same place.
\begin{tabu} {X<{\hspace*{.5em}}>{\hspace*{.5em}}X}
    \field{thing}& \hrulefill\\
    \blank & \hrulefill\\
    \field{thing}& \hrulefill\\
\end{tabu}

\bigskip

% all blanks flush with each other at right end of 1st column.
\begin{tabu} {X @{\hspace{\dimexpr1em+2\tabcolsep}} X}
    \field{thing}& \hrulefill\\
    \blank & \hrulefill\\
    \field{thing}& \hrulefill\\
\end{tabu}

\end{document}

The black boxes are apparently due to a wrong definition for \field.