Aberrant vertical spacing with {align} and friends?

Question

Consider documents with the following preamble:

\documentclass[12pt]{article}
\usepackage{amsmath}

Question #1.

Why do we get bad alignment with the following code?

\noindent
$-3x$\\[-\baselineskip]%
$\begin{aligned}\!
    \phantom{-3x} (x+1) & = 4 \\ & = 2^2  
\end{aligned}$

I cannot figure out an appropriate amount of negative space to use in order to achieve alignment between the inline math and the aligned environment, using any plausible combination of \baselineskip and \jot. (The dimension \dimexpr-\baselineskip+0.5\jot almost works, for reasons which are opaque to me, but still doesn't do the job.)

Question #2.

What is the correct spacing to achieve the proper alignment in this case, independent of font size etc.?

---

Background

I'm puzzling over how to produce a robust solution to this problem on equation alignment, and have come to the conclusion that I don't understand what LaTeX is doing with vertical spacing when it comes to aligned. I would like to describe how to undo one line of vertical space, but this does not appear to be straightforward when the thing on the next line is a multiline equation.

---

Test cases which work properly

The following are my little tests to try to asses what is going on with the spacing. The latter two cases only served to make the failure in the above example extra confusing.

1. Undoing vertical spacing of two lines in textmode: easy. The following code illustrates roughly the effect that I would like to achieve:

   \noindent
   Testing \phantom{testing} 1 \phantom{2} 3 
   \\[-\baselineskip]%
   \phantom{Testing} testing \phantom{1} 2 \phantom{3}
   

   This appears as a single line of text, saying "Testing testing 1 2 3" with appropriate alignment and spacing, as it should.

2. Undoing vertical spacing involving math and an aligned environment: nothing inherently changes about the spacing.

   \noindent
   $-3x$
   \\[-\baselineskip]%
   $\begin{aligned}\!
       \phantom{-3x} (x+1)    
   \end{aligned}$
   

   This gives output which looks like −3x(x+1) with appropriate alignment and spacing.

3. Multiline equations without additional vertical spacing: If we make the aligned environment into a multiline equation — but remove the vertical spacing between the two lines of the equation — then the vertical alignment still functions properly.

   \noindent
   $-3x$
   \\[-\baselineskip]%
   $\begin{aligned}\!
       \phantom{-3x} (x+1) & + 4 \\[-\baselineskip] & + 4  
   \end{aligned}$
   

   I must admit that I don't understand why I don't have to involve the unit \jot to make this work properly, but once again we obtain proper vertical alignment, with something that looks like −3x(x+1)+4.

I would like to see a way of making this work in an obvious and portable way, regardless of what other things are going on. Naturally, if I were using align instead of aligned, I might also have to involve \abovedisplayskip, which I'm comfortable with; though if there are "gotchas" in the spacing beyond that, I'd be again a little annoyed. But in any case, I think the first step is to understand how it's supposed to work with an environment which lives inside of mathmode, such as aligned.

Answer 1

If you add \showoutput, you will see

...\hbox(10.14995+4.35004)x390.0, glue set 368.11467fil []
...\glue -13.5
...\penalty 300
...\glue(\lineskip) 1.0
...\hbox(19.0+12.99998)x390.0, glue set 305.5215fil []

which tells you the extra vertical space is \lineskip glue. As Andrew Swan's answer suggests, you could hide the height which would stop lineskip glue being used, but that has the danger of over-printing. An alternative is to use \strut-s to ensure baseline spacing, and then compensate for lineskip glue as well as baselineskip:

\documentclass[12pt]{article}
\usepackage{amsmath}

\begin{document}

Question \#1.

\showoutput
\showboxdepth3

Why do we get bad alignment with the following code?

\noindent
$-3x$\strut\\[\dimexpr-\baselineskip-\lineskip\relax]%
\strut$\begin{aligned}\!
    \phantom{-3x} (x+1) & = 4 \\ & = 2^2  
\end{aligned}$

\end{document}

Answer 2

Rewritten answer

The basic definition of aligned without an optional argument puts its material in a \vcenter box. \vcenter places its contents vertically centered with respect to the axis of the formula, this lies higher than the baseline. If you don't start a new line, then you will see the material vertically centered in this way.

When you start a new line then LaTeXs first inclination is to add vertical skip for the baseline, but then it looks to see whether the subsequent line does not overlap with the previous one. If the top of the next line is within \lineskiplimit of the bottom of the previous line, then \baselineskip will be ignored and \lineskip with be added between the lines, i.e. between the bottom of the first line and the top of the second.

In your case the aligned environment (with 2 or more rows) on the next line is high enough that this comes in to effect, and so the spacing is a \lineskip before the top of the environment, within in the environment the material is such that the baseline of the first line is \baselineskip below the top. Thus backing up \baselineskip is not sufficient and as David Carlisle explains. In your test cases 2 and 3, the aligned material is no more than a line high, so the \lineskip does not kick in.

A point to note is that aligned inserts struts to make lines uniformly high. The consequence of this is that a single line in an aligned with normal height/depth ends up aligned with the baselines anyway.

If you give aligned the [t] option, then it switiches to \vtop instead of \vcenter. This sets the baseline of the alignment to the baseline of the first line in the alignment and suits your purposes better.

The sample below demonstrates this and compares \vtop and \vcenter with aligned:

\documentclass{article}

\usepackage{amsmath}

\setlength{\parindent}{0pt}
\setlength{\parskip}{10pt}

\begin{document}

\verb+aligned+: no newlines

$3x$%
$\vcenter{\hbox{$a$}}$%
$3x$%
$\vcenter{\hbox{$\strut a$}}$%
$3x$%
$\begin{aligned}a\end{aligned}$%
$3x$%
$\begin{aligned}u\\v\\w\end{aligned}$

\verb+aligned+: with newlines

$3x$\\[-\baselineskip]%
\hspace*{1.1em}$\vcenter{\hbox{$a$}}$\\[-\baselineskip]%
\hspace*{1.7em}$3x$\\[-\baselineskip]%
\hspace*{2.8em}$\vcenter{\hbox{$\strut a$}}$\\[-\baselineskip]%
\hspace*{3.4em}$3x$\\[-\baselineskip]%
\hspace*{4.5em}$\begin{aligned}a\end{aligned}$\\[-\baselineskip]%
\hspace*{5.5em}$3x$\\[-\baselineskip]\\[-\baselineskip]%
\hspace*{6.6em}$\begin{aligned}u\\v\\w\end{aligned}$

\verb+aligned[t]+: no newlines

$3x$%
$\vtop{\hbox{$a$}}$%
$3x$%
$\begin{aligned}[t]u\\v\\w\end{aligned}$

\verb+aligned[t]+: with newlines

$3x$\\[-\baselineskip]%
\hspace*{1.1em}$\vtop{\hbox{$a$}}$\\[-\baselineskip]%
\hspace*{1.7em}$3x$\\[-\baselineskip]%
\hspace*{2.8em}$\begin{aligned}[t]u\\v\\w\end{aligned}$

\end{document}

However, if you have high material on the first line of your aligned, starting a newline is going to be harder to control, even with the [t] option: the baseline is now moved down by the height of that line plus \lineskip (plus the depth of the previous line). In some situations you may use \smash on any tall vertical material in the first line of the aligned. The examples below have exaggeratedly high material introduced by a \vrule.

\documentclass[12pt]{article}
\usepackage{amsmath}
\begin{document}
\noindent
$-3x$\\[-\baselineskip]
$\begin{aligned}[t]
    \phantom{-3x} \smash{\vrule height 30pt (x+1)} & = 4 \\ & = 2^2
\end{aligned}$

\noindent
in contrast to:

\noindent
$-3x$\\[-\baselineskip]
$\begin{aligned}
    \phantom{-3x} \vrule height 30pt (x+1) & = 4 \\ & = 2^2
\end{aligned}$

\noindent
$-3x$\\[-\baselineskip]
$\begin{aligned}[t]
    \phantom{-3x} \vrule height 30pt (x+1) & = 4 \\ & = 2^2
\end{aligned}$
\end{document}

In general, you will want to introduce a \vphantom in the proceeding material to allow vertical space for the high material.

However, why you are starting a new line is a mystery to me:

\documentclass[12pt]{article}
\usepackage{amsmath}
\begin{document}
\noindent
$-3x$%
$\begin{aligned}[t]\!
  \vrule height 30pt (x+1) & = 4 \\ & = 2^2  
\end{aligned}$

\end{document}