# Understanding \needspace

My recent question about How to get "nonbreaking vertical space"? is mostly solved, except that it looks as though the use of \needspace doesn't agree with what the layout engine does.

That is, I use \needspace and everything looks like the item should fit, but it still gets moved to the next page. This is confirmed by both looking at current position and space remaining, and even eyeballing -- a two-inch object should easily fit when there are five inches available in the column before the page break.

To try to solve this, I need to make sure I understand what \needspace does, and how.

\newcommand{\needspace}[1]{%
\begingroup
\setlength{\dimen@}{#1}%
\vskip\z@\@plus\dimen@
\penalty -100\vskip\z@\@plus -\dimen@
\vskip\dimen@
\penalty 9999%
\vskip -\dimen@
\vskip\z@skip % hide the previous |\vskip| from |\addvspace|
\endgroup
}


If I read this correctly, the macro:

1. stores the parameter (must be a length).
2. skips down that distance (\z@\@plus\dimen@... as a rubber length?)
3. slightly encourages the current location for a break (negative penalty... TeX doesn't need to put a break here, as \newpage would force, but the \penalty- 100 says it's okay) and then skips back up the original distance.

So far, I think I see what's happening. Then it does almost the same thing:

1. skips down the requested distance.
2. very strongly (\penalty 9999, almost a "do not break here ever") discourages a break.
3. skips back up.
4. "zero skip" to give the "skip cleanup" something safe to eat.

... and now I'm not so sure I understand what's happening.

The first part makes sense to me. Move down the requested amount, probe to see if this is a good place to break (negative penalty; if it's a good place it'll break), then move back up. If there was no break, we end up back where we started and the requested space should be available. If there was a break, that \vskip really does nothing.

Why do it the second time?

Exploration 1: MWE

I now have an MWE to use in exploring the topic.

\documentclass[letterpaper,10pt,twoside,twocolumn]{article}
\usepackage{lipsum}
\usepackage{tcolorbox}
\usepackage{fp}
\usepackage{needspace}
\usepackage[top=1in,bottom=1in,right=1in,left=1in]{geometry}

\makeatletter
\newlength\egd@nsc@space
\newlength\egd@nsc@needed
\newsavebox\egd@nsc@box

\newenvironment{needspacecalc}[1][0pt]{
\setlength{\egd@nsc@space}{#1}
\begin{lrbox}{\egd@nsc@box}
}{
\end{lrbox}
\setlength{\egd@nsc@needed}{\dimexpr\ht\egd@nsc@box+\egd@nsc@space\relax}
\marginpar{\the\pagetotal\quad\the\pagegoal\quad\the\egd@nsc@needed}
\needspace\egd@nsc@needed
\noindent\usebox\egd@nsc@box
}
\makeatother

\newcommand{\testbox}{
\medskip\par
\begin{needspacecalc}[1.5\baselineskip] % "need 1.5 lines after the box
\begin{tcolorbox}
TColorBox stuff goes in here.
\vskip4\baselineskip\mbox{}
\end{tcolorbox}
\end{needspacecalc}
\medskip\par
}

\raggedbottom

\title{Suppressing Page Breaks MWE}
\begin{document}
\maketitle
\lipsum[4]
\lipsum[4]
\lipsum[4]
\lipsum[4]
\testbox
\lipsum[4]
\bigskip
\lipsum[5]
\testbox
\lipsum[5]
\end{document}


The needspacecalc environment stores its contents in an lrbox, measures the height of that box, adds the requested amount to this height, then invokes \needspace before outputting the lrbox. (This goes back to my original question, but it turns out handy to use here.)

The \marginpar in the needspacecalc wouldn't be here in production, but it's good to see what's happening.

When I run this as written, I get mostly what I would expect. At the bottom of the left column I'm at position (all truncated) 485pt of 528pt available, I want 104pt, so there clearly isn't room and my block goes to the next column. At the bottom of the right column I'm at position 410pt of 528pt, I want 104pt, and... the block gets moved to the next column. Maybe because 410+104 = 514, which is kind of close...? In any case, it got moved.

If I change \needspace like this

    \penalty -100\vskip\z@\@plus -\dimen@
%\vskip\dimen@
%\penalty 9999%
%\vskip -\dimen@
\vskip\z@skip % hide the previous |\vskip| from |\addvspace|


I get the same result. I kind of expected this because it looks like the commented-out instructions don't do anything that wasn't already done... but maybe they do something that doesn't apply here.

However, regardless of whether \needspace has those three lines present or commented out, if I change \testbox so it doesn't actually use the needspacecalc environment at all

\newcommand{\testbox}{
\medskip\par
%\begin{needspacecalc}[1.5\baselineskip] % "need 1.5 lines after the box
\begin{tcolorbox}
TColorBox stuff goes in here.
\vskip4\baselineskip\mbox{}
\end{tcolorbox}
%\end{needspacecalc}
\medskip\par
}


the second block does appear at the bottom of the column, with two lines of text after it. There clearly was room for it, but it only got used if I didn't try to reserve it first.

(If you're confused, and it's not because of my writing, you're not alone....)

• Similar, but not the same. If I read this right, make a 0pt-width ('invisible') rule the length I want to reserve, go back up, go up one more line, then add a newline... this did not look right when I ran it. Everything ended a bit too high. – Keith Davies Jan 17 '17 at 6:22
• Compiling your MWE with \tracingpages=1 shows that, for the second \textbox, the ~105pt of stretchability inserted by \needspace, plus the 10pt already accumulated on the page, combine with the \penalty -100 to yield a net cost of 7, that turns out to be more appealing than the cost (150) of the break that ranks second, which, indeed, would occur after the first line of the paragraph that follows the \testbox itself, at t=528.72256, with b=0; this cost of 150 is thus entirely due to the value of p, which, in turn, comes from the \clubpenalty, – GuM Mar 19 '17 at 1:39
• I understood every single word of that comment... but will need to review breaking rules to understand the entirety :) – Keith Davies Mar 19 '17 at 21:38
• Thank you, indeed I tried to make every single word meaningful! (:-) Essentially, my comment boils down to this: the implementation of the \needspace command seems defective, if not buggy. I too do not understand that maneuver of probing for the availability of the required space first as a stretch component, then as a natural length, but with an extremely high penalty. Maybe \vskip\z@\@plus\dimen@ \penalty200 \vskip\z@\@plus-\dimen@ would be better. Allow me to say that I cannot trust thoroughly a person who puts a % sign after 9999 (unless he does so to save memory…). (;-) – GuM Mar 19 '17 at 23:43
• "the implementation of the \needspace command seems defective, if not buggy"... at one point -- and this is what prompted the entire topic -- I suspected that might be the case. I am always hesitant to announce that conclusion about commonly-used code or published library unless I understand it well enough to prove the fault. At this point I'm not entirely certain, but you've given me a hint as to where to look. – Keith Davies Mar 20 '17 at 0:02

## 3 Answers

At first the main assumption is wrong: \vskip 0pt plus \dimen@ is very different from \vskip \dimen@. The first is a 0pt space than can (but doesn't need to stretch).

Now lets try what happen if one remove some parts of the definition:

\documentclass[]{book}
\makeatletter
\newcommand{\needspace}[1]{%
\begingroup
\setlength{\dimen@}{#1}%
\vskip\z@\@plus\dimen@
\penalty -100
\vskip\z@\@plus -\dimen@
%    \vskip\dimen@
%    \penalty 9999%
%    \vskip -\dimen@
\vskip\z@skip % hide the previous |\vskip| from |\addvspace|
\endgroup
}
\begin{document}
Without the second vskip needspace doesn't do anything:

\needspace{100cm}

blbl
\end{document}


\documentclass[]{book}
\makeatletter
\newcommand{\needspace}[1]{%
\begingroup
\setlength{\dimen@}{#1}%
%   \vskip\z@\@plus\dimen@
%    \penalty -100
%    \vskip\z@\@plus -\dimen@
\vskip\dimen@
\penalty 9999%
\vskip -\dimen@
\vskip\z@skip % hide the previous |\vskip| from |\addvspace|
\endgroup
}
\begin{document}
Without the first vskip needspace works as expected:

\needspace{100cm}

blbl
\end{document}


# But

one get a message in the log:

Underfull \vbox (badness 10000) has occurred while \output is active []


as the break point is before the \vskip (before glue is normally a breakpoint)

\documentclass[]{book}
\makeatletter
\newcommand{\needspace}[1]{%
\begingroup
\setlength{\dimen@}{#1}%
\vskip\z@\@plus\dimen@
\penalty -100
\vskip\z@\@plus -\dimen@
\vskip\dimen@
%    \penalty 9999%
\vskip -\dimen@
\vskip\z@skip % hide the previous |\vskip| from |\addvspace|
\endgroup
}
\begin{document}
Without the second penalty needspace doesn't work:

\needspace{100cm}

blbl
\end{document}


In this case the both \vskip\dimen@ cancel each other out before TeX start to consider a page break.

## Conclusion

You need the \vskip 0pt plus stretch to fill the first page and avoid an underful page

You need the \vskip \dimen@ to get the wanted page break

You need the first penalty as break point

You need the second penalty to avoid that the two vskips cancels each other.

• Now I have thoroughly understood your answer, and I can upvote it! (:-) I think that adding the following remarks would help others to understand more quickly: 1. TeX doesn’t make any decision about where to cut off a page until a legal breakpoints comes along having either (a) associated penalty ≤ -10000, or (b) infinite cost, which essentially means that the page-so-far has become overfull (_The TeXbook, p. 112). – GuM Apr 10 '17 at 22:13
• 2. The \vskip\dimen@ (i.e., the second one) only serves the purpose of making condition (b) occur, when the requested space exceeds the space remaining on the page. 3. The ensuing \penalty 9999 is there only to provide the “legal breakpoint” that triggers the page cost computation, at a time when the \vskip\dimen@ has not yet been canceled out. – GuM Apr 10 '17 at 22:19
• 4. (If and) when, in step 3, TeX is forced to make a decision about where to break the page, it will most likely choose the previous \penalty -100 (unless something even better came before), thus adding \dimen@ to the current \pagestretch; it will never choose the \penalty 9999 as the breakpoint (as you have already remarked). 5. These remarks are easily verfied by activating \tracingpages. – GuM Apr 10 '17 at 22:26
• … And in the end: great answer!! :-) – GuM Apr 10 '17 at 22:32
• Heh, I read this answer yesterday and thought I'd have to give it some thought to understand... and then saw ShreevatsaR's answer and it all came together. – Keith Davies Apr 11 '17 at 14:47

A few months ago I looked at needspace, and also read the wonderful paper Breaking Paragraphs into Lines, so sharing what I understand (before I forget everything).

To recall: \needspace{length} is meant to be used when we want to say: "the next [length] of material should stay together and not get split over a page break". For concreteness, let's assume that we call \needspace with an argument of 42pt (say). Then, from looking at the definition (included in the question) you can see that it results in the following sequence of glue (skips) and boxes:

• \vskip 0pt plus 42pt (glue of ideal length 0pt and stretchability 42pt)
• \penalty -100 (a negative penalty, suggesting a good place to break)
• \vskip 0pt plus -42pt (glue of ideal length 0pt and stretchability -42pt)
• \vskip 42pt (glue of exactly 42pt, no stretch or shrink)
• \penalty 9999 (large penalty)
• \vskip -42pt (glue of exactly -42pt)
• \vskip 0pt (this one is just for macros that look at \lastskip, and can be ignored if you're not using such macros nearby)

To make sense of this sequence of items, there are a few things you need to know:

• By the rules of TeX breaking, among the above items, a break is allowed only immediately after a penalty (either the -100 or the 9999 one), or after the first glue if there was a box (non-discardable item) preceding it.

• After TeX inserts a break, it discards all the penalties and glue that immediately follow.

• For page breaks (unlike for line breaks in a paragraph), TeX does not use a global algorithm ("optimum-fit"), but makes the best decision for each page locally ("best-fit"). In more detail: each time TeX adds items (think: lines) to a page, it computes the cost (using badness and penalty) of breaking the page there. When the page gets overfull or when a penalty of ≤-10000 is encountered, TeX knows it can't go on piling more stuff onto the page, and breaks the page at the best (least-cost) place seen so far. (Whatever remains becomes material for the next page.)

So, for the above list of items, these are the four allowed possibilities of page breaks, and what would happen in each case:

• ### No page break

In this case, with no breaks, the penalties don't matter, and the glues neatly cancel each other out. (This is important, and nothing weird is introduced.)

• ### Page break after the first \vskip

This will never happen (even if allowed), because it is strictly worse than breaking after the immediately following \penalty -100: in that case (compared to this one), TeX would gain both 42pt of additional stretchability on the first page (which decreases badness) and the negative penalty.

• ### Page break after the \penalty -100

In this case, 42pt of extra stretchability gets added to the first page (so that it can be ragged bottom if necessary), while on the new page all the leading glue is deleted and so the rest of the items are ignored.

• ### Page break after the \penalty 9999

This too will never happen (despite being just barely allowed), again because it is strictly worse than breaking after the \penalty -100, for similar reasons. (Exercise: prove or come up with a counterexample.)

Fine, so we can break at the \penalty -100 or not, but how does our list of items achieve the goal of needspace: of not having a break within the next (approximately) 42pt? Well, consider the two cases:

• # The page has (much) more than 42pt remaining

(That is, more than 42pt can be squeezed in, without the page becoming overfull.) In this case, TeX will consider the two or three legal breakpoints from the above list, make note of their costs (the one at \penalty -100 will be best), and keep going, and probably end up breaking the page where it would have normally done. This is because by breaking at the \penalty -100 it would gain a bit in stretchability (decreasing badness) and in penalty, at the cost of having to stretch the page by the remaining length of at least 42pt (increasing badness). (This cost may overcome the gains, but this is not guaranteed, which is why needspace says "approximately", and which is why you're seeing what you're seeing.) So probable outcome: the "no page break" case above (but the "break at the \penalty -100 case also remains somewhat likely, which is the direction in which needspace errs).

• # The page has less than 42pt remaining

In this case, once TeX has seen the \vskip 42pt (the fourth item in the list), the page has become overfull, so as soon as TeX sees the next \penalty 9999 (an allowed breakpoint, though a very bad one) it picks the best break seen so far. Probable outcome: the "page break after \penalty -100" case above, or just possibly some even earlier break.

I hope that now you can see why every item in the sequence is necessary: the actual "force a page break if there's not 42pt of room left" is achieved by the \vskip 42pt\penalty 9999, the \penalty -100 earlier is intended to give a good breakpoint for doing this (if necessary), the \vskip 0pt plus 42pt before it is added to give enough stretch to the first page, and the others (negative height and stretch) are added to cancel out the effect of these if the break isn't taken.

You can see a trace of these deliberations by compiling your document with \tracingpages=1 and looking in the log file.

# Advertisement

Although this case above turned out to hinge on the greedy algorithm used for page breaking, in spirit it is similar to many elegant solutions to typesetting problems in the Knuth-Plass Breaking Paragraphs into Lines paper—author lines, ragged right and centered text, typesetting source code or an index at varying widths—all solved by such "magical" sequences of boxes, glue and penalties. (These are also in The TeXbook as double dangerous-bend exercises and in Appendix D: Dirty Tricks, but it's more expository in the paper.) After showing off these solutions, there is some "algebra" that helps you come up with similar constructions yourself. The whole paper is an enjoyable read. The most updated version (with notation closer to TeX's, such as using ~ for ties rather than &) occurs in the book Digital Typography, but you can also find the earlier version online and the beautiful ideas are all there.

Edit:

1. Armed with this understanding, we can tweak the macro to suit our preferences better. For example, if we don't like breaking unnecessarily on the first page:

• why a penalty of -100? A page break is already guaranteed if the required length cannot fit. By taking this break TeX would already gain stretchability; why give it a further boost with a negative penalty? So you can try changing the penalty to 0 or even a slightly positive value.
• why add so much stretchability to the first page, making that break so desirable? The stretchability doesn't need to be exactly the same length as the dimension passed in to \needspace; you can decrease it. And if you want the pages flush bottom, you can even delete it altogether and have TeX stretch the first page as necessary.

In your MWE, where \needspace is called with 104.86pt, the break (before the second block) is avoided if we increase the penalty from -100 to 43 (keeping stretchability the same), or decrease the stretchability from 104.86pt to ≤92.81pt (at penalty 0) or to ≤76.7pt (keeping penalty -100).

2. [Insert rant about the phenomenon of "there's a package for that" rather than empowering users by giving them information so they can come up with solutions themselves, making the system less mysterious. But usually the former will win; this is not specific to LaTeX.]

3. You may not need needspace! If you're computing the length of some material (a box or paragraph or whatever) that you'd like to avoid breaking, and using \needspace with that length, you can probably achieve better results by adding an appropriate sequence of glue and penalties after it.

• Okay, now I get it! I could see what was happening (and that I need to correct the question text to reflect that), but I didn't understand why, all the implications of it. Upvoting answer for now, will likely accept as answer after I update the question. – Keith Davies Apr 11 '17 at 14:38
• @KeithDavies: What happens in your question is that \vspace\dimen@\penalty9999 does not cause the page to become overfull (it has t=515.72256 plus 10.0 minus 8.0), so TeX continue to look ahead for better breaks. Now, the break at the \penalty-100 has c=7#; then TeX finds: (1) break at \penalty9999, with c=10210; (2) break between the tcolorbox and the next (empty) line, with b=1990 p=300 c=2290 (the p=300 is the sum of \clubpenalty and \widowpenalty); (continues) – GuM Apr 11 '17 at 16:44
• (continued) (3) break before the following \lipsum paragraph, with b=425 p=0 c=425; (4) break after the first line of this paragraph, with b=0 p=150 c=150 (p=150 is from \clubpenalty); (5) break after the following line, with b=* p=100 c=* (p=100 from \brokenpenalty): now the page has become overfull, so TeX decides where to cut it off. It turns out that break (4) would be the best one (b=0, with \pagetotal practically equal to \pagegoal), if it weren’t for the \clubpenalty (continues). – GuM Apr 11 '17 at 16:57
• (Continued) Actually, the break at the \penalty-100 is still the best remembered breakpoint. Note that, in this particular case, removing the \vskip\dimen@\penalty9999\vskip-\dimen@ doesn’t affect the outcome, because, by pure chance, the page-so-far gets overfull while the \penalty-100 is still the best one, as said. This may happen every time the space required with \needspace gets very close to, but not quite up to, causing the current page to become overfull. (Finished! :-) – GuM Apr 11 '17 at 17:26
• If I could, I would upvote again just for the “rant about the phenomenon of ‘there's a package for that’…"! – GuM Apr 11 '17 at 18:26

This is how I reserve space. When \vspace is used in the middle of a line, it waits until the end of the line to be implemented. Note that \vspace is ignored at the top or bottom of a page. So if the \rule extends too far, it either winds up on the next page or the \vspace is ignored (not sure which).

\documentclass[letter]{article}
\usepackage{showframe}
\usepackage{lipsum}

\newcommand{\needspace}[1]{\rule{0pt}{#1}\vspace{-#1}\vspace{-\parskip}\par}

\begin{document}
\rule{1pt}{40\baselineskip}% 41 will force a break.

\needspace{5\baselineskip}
\lipsum[1]
\end{document}