# Is there a way to get at the value of \prevdepth from the main vertical list inside the output routine?

To motivate that (admittedly) strange question:

When TeX builds a vertical list it keeps track of the depth of the last box added in a special variable called \prevdepth which is used to calculate baselineskip glue for paragraph lines. When a new vertical list is being built (and the OR starts a new vertical list) TeX reinitializes \prevdepth and when processing ends it resets it to the old (outer) value.

With normal output routines (that ship out pages) that's okay as the \prevdepth (after the output routine ends) will match the last box on the main vertical list if there is a remainder that was not put into box255. If the main vertical list is empty the value is wrong but that doesn't make a difference because TeX will use \topskip on top of the first line so it is essentially ignored.

But the situation is different if the output routine puts material back to be reprocessed. In that case the last box on the main vertical list is a) either the box that was not used in the OR and that has a depth matching \prevdepth or it is the last box from the stuff that the OR contributed (which may have a different depth).

So to make baselineskip calculations work correctly we would need to know which situation we are in: has everything been packaged into box255 or is there an unused remainder? If that question could be answered that would also solve the issue but this is to my knowledge not possible.

An alternative solution would be to pretend that the material generated by the OR has the same depth as the outer \prevdepth but for this I would somehow need to know its value. Knowing its value is simple if we force the OR with an explicit penalty because then I can just save it and used the saved value. But what can be done if the OR is called by TeX on its own?

That's really the question above ... any idea (as sneaky as it needs to be) to somehow get at this value in this particular case?

Or alternatively, any method to clearly determine whether or not the main vertical list is emptied.

Please note that I'm looking for a solution that works with all TeX engines, so doesn't make use of, say, lua programming.

To motivate a bit further, here is a short plain TeX file that shows the general problem (though here it could be fixed as we explicitly force the output routine):

\tracingonline=1

test with g to get a depth

\showthe\prevdepth % we see the prevdepth from the last line

% now assume we have some OR that traps the data and does something with it
% afterwards it is pushing back new or changed data t build pages, In the example I
% simply dropped the collected data and replaced it with soemthing else which has a different
% \prevdepth. A real life example would be rebalancing existing material.

\output{\setbox0=\vbox{\unvbox255}%
next line will be too close\par
\showlists
\showthe\prevdepth
}
\eject

% now we see that the prevdepth should be (and is 0pt) last line jiust contains
% characters without depth

\showlists
\showthe\prevdepth

% but now we got the old \prevdepth back even though it is no longer valid, as
% the OR simply pops the nest even though it is no longer valid.

% As a result we will get the wrong alignment on the next paragraph

Second line \par
\showlists

%  here we can see that  we are off by the 1.9...pt prevdepth as we should see
% 12pt baseline to baseline but  we don't

\output{\plainoutput}

\bigskip

But we really should see:

Next line will not be too close \par Second line

\bye


If we run this we get:

and if we look into the log we can also see how the wrong \prevdepth is messing up vertical spacing:

> 1.94444pt.
l.6 \showthe\prevdepth
% we see the prevdepth from the last line
?

### internal vertical mode entered at line 18 (\output routine)
\hbox(6.94444+0.0)x469.75499, glue set 337.7548fil
.\hbox(0.0+0.0)x20.0
.\tenrm n
.\tenrm e
.\tenrm x
.\tenrm t
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm l
.\tenrm i
.\tenrm n
.\tenrm e
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm w
.\tenrm i
.\tenrm l
.\tenrm l
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm b
.\kern0.27779
.\tenrm e
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm t
.\tenrm o
.\kern0.27779
.\tenrm o
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm c
.\tenrm l
.\tenrm o
.\tenrm s
.\tenrm e
.\penalty 10000
.\glue(\parfillskip) 0.0 plus 1.0fil
.\glue(\rightskip) 0.0
prevdepth 0.0, prevgraf 1 line
### vertical mode entered at line 0
### recent contributions:
\penalty 10000
prevdepth 1.94444, prevgraf 1 line

! OK.
<output> ...line will be too close\par \showlists
\showthe \prevdepth }
\break ->\penalty -\@M

l.18   \eject

?
> 0.0pt.
<output> ...se\par \showlists \showthe \prevdepth
}
\break ->\penalty -\@M

l.18   \eject

?

### vertical mode entered at line 0
### current page:
\glue(\topskip) 3.05556
\hbox(6.94444+0.0)x469.75499, glue set 337.7548fil
.\hbox(0.0+0.0)x20.0
.\tenrm n
.\tenrm e
.\tenrm x
.\tenrm t
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm l
.\tenrm i
.\tenrm n
.\tenrm e
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm w
.\tenrm i
.\tenrm l
.\tenrm l
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm b
.\kern0.27779
.\tenrm e
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm t
.\tenrm o
.\kern0.27779
.\tenrm o
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm c
.\tenrm l
.\tenrm o
.\tenrm s
.\tenrm e
.\penalty 10000
.\glue(\parfillskip) 0.0 plus 1.0fil
.\glue(\rightskip) 0.0
\penalty 10000
total height 10.0
goal height 643.20255
prevdepth 1.94444, prevgraf 1 line

! OK.
l.23   \showlists

?

> 1.94444pt.
l.24   \showthe\prevdepth

?
### vertical mode entered at line 0
### current page:
\glue(\topskip) 3.05556
\hbox(6.94444+0.0)x469.75499, glue set 337.7548fil
.\hbox(0.0+0.0)x20.0
.\tenrm n
.\tenrm e
.\tenrm x
.\tenrm t
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm l
.\tenrm i
.\tenrm n
.\tenrm e
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm w
.\tenrm i
.\tenrm l
.\tenrm l
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm b
.\kern0.27779
.\tenrm e
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm t
.\tenrm o
.\kern0.27779
.\tenrm o
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm c
.\tenrm l
.\tenrm o
.\tenrm s
.\tenrm e
.\penalty 10000
.\glue(\parfillskip) 0.0 plus 1.0fil
.\glue(\rightskip) 0.0
\penalty 10000
\glue(\parskip) 0.0 plus 1.0
\glue(\baselineskip) 3.11111
\hbox(6.94444+0.0)x469.75499, glue set 400.31046fil
.\hbox(0.0+0.0)x20.0
.\tenrm S
.\tenrm e
.\tenrm c
.\tenrm o
.\tenrm n
.\tenrm d
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm l
.\tenrm i
.\tenrm n
.\tenrm e
.\penalty 10000
.\glue(\parfillskip) 0.0 plus 1.0fil
.\glue(\rightskip) 0.0
total height 20.05556 plus 1.0
goal height 643.20255
prevdepth 0.0, prevgraf 1 line

! OK.
l.32     \showlists

• too bad you didn't pose this question a year ago, when it could have been sent to don as part of his scheduled cleanup. – barbara beeton Sep 14 '14 at 12:23
• @barbarabeeton he wouldn't have touched it I guess, well he could have by adding a new primitive. But the current behavior is not really a bug, it works in cases where Don wanted it to work :-) ... there is a TUB article coming up on this, but when I wrote it last night the above remained as an open question. – Frank Mittelbach Sep 14 '14 at 12:36
• i'm sure he wouldn't have changed anything, but he might have made a suggestion. looking forward to the article -- thanks for the heads up. – barbara beeton Sep 14 '14 at 12:38
• So we assume if Frank is really stuck on a TeX related problem he will get the anser on TeX.sx. I'm marking this question as favorite just to whitness this doomsday scenario. :-) – Bananguin Sep 14 '14 at 20:48
• @MartinSchröder if it has been solved then show me, I would certainly be interested to learn about it, otherwise that's a fairly empty statement. However, even if it has (which I doubt to be honest) I'm really much more interested in finding a solution that works in all TeX engines not just in one. – Frank Mittelbach Sep 16 '14 at 5:56

@wipet in his answer has shown how to resolve the problem as long as we can assume that that the \prevdepth of all material in the document is sufficiently small (that it lies below \maxdepth in fact). In that case we can use the depth of box 255 as a measure for the \prevdepthcalculations that will have taken place if there is any remainder in recent contributions, and use this to adjust the new depth to match that. An if there isn't any remainder then this doesn't really matter either. This trick is in fact already mentioned by Don Knuth in the TeXbook where he discusses an output routine that add index headings in random places in the text (and normal text has this nice property of having a depth smaller than \maxdepth ... normally but unfortunately not always.

However, this stops working if this can't guaranteed and the general case is what I was and am after. He is also correct in stating that finding out what the value of \prevdepth on the main vertical list is isn't really going to help us as such, so my question above was partly incorrect: we also need to know if there is in fact something on the recent contributions and if so deal with it.

So what I tried to get my head around over the last days if that is really impossible to find out within base TeX (or if it needs an extension in luatex or is already available there and resolved in ConTeXt ... would be interesting to learn about it beside that claim @Martin you made). And as it stands now it is possible in TeX after all. The solution is a bit complex and perhaps it can be simplified further but it is not so complex that it can't be used (and hopefully not too complex that I overlooked some cases).

The main idea is not to try to determine the situation directly but instead use 2 output routines in succession that just ensure that the recent contributions are empty. Along the way we gather enough information to afterwards dissect the material gathered and do whatever we like.

My initial idea of using \aftergroup to regain control and have a look doesn't work because the token inserted that way will not be executed when the output routine ends. Instead TeX immediately calls again on a procedure "buildpage" that takes anything on recent contributions and moves it to the main vertical list and only once that has happened my inserted token is looked at (in other words too late).

So the more complicated approach is that the first OR puts box255 back but only it has changed \vsize to the largest possible dimen. Additionally it uses \aftergroup so that we gain control later again. As we have changed \vsize we will get everything including the material recent contributions onto the main vertical list and only then our control token kicks in. Finally we change the output routine to a second one and then return.

The token inserted by \aftergroup will then issue a forcing penalty (actually it does a little more, see below) so that everything is grabbed and the second OR is called.

Inside that OR we are now in a better situation:

• we know that recent contributions is empty (except for a (\penalty 10000)
• we can store away what was gathered in box255
• or we could manipulate it using \vsplit etc to get, for example the amount split of that we would have gotten in the first OR
• and we can use \aftergroup to gain control after the OR has ended.
• the latter allows us to change \prevdepth on the main vertical list to represent whatever is needed

And that should (I believe) do the trick completely.

Here is the (more or less documented code including a bit of test data to play out various scenarios). It is a bit longish but largely because I tried to properly document the most important aspects and some of the more subtle points. Enjoy:

\tracingoutput1
\tracingpages1
\tracingonline1

\vsize20\baselineskip

\lineskip=13pt  % for identifications

% this is our trial material used below. We will arrange things so
% that the first para will be longer than a page so that we will end
% up with some material on recent contributions. The OR is actually
% then triggered when the first p'' is seen. Alternatively one can
% uncomment the \vskip or the \penalty in which case the OR will be
% triggered by them or you could uncomment Line2 and3'' then the
% break happens somewhere in the middle of the first paragraph (in
% vmode inn that case)

\def\testmaterial{%
Line 1 \hfil\break
%  Line 2 \hfil\break
%  Line 3 \hfil\break
some text some text some text some text some text some text
some text some text some text
and some more text ggg \vrule depth 88pt

%\ vskip 17pt
% \penalty 15

pppppppppppppppppppppppppp

\showlists
}

% now this here is just to see the whole stuff being processed by the
% normal OR and see the \showlists result for it. One can then compare
% that to the showlists result we get later to check for differences

\testmaterial

\vfill

\eject

%==============================================

\newdimen\savedvsize
\newbox\savedORbox

% now for a set of special output routines:
% the first one does the following
%
% - save away current \vsize and set it to \maxdimen
% - then unbox 255 and readd the output penalty (unless it is 10000)
% - set up a new output routine for the next time
% - finally install control with \aftergroup\addendpenalty
%

% The point here is that the \aftergroup token is not actually
% directly executed the moment the OR ends. If TeX ends an OR it looks
% at the recent contribution and if they are not empty it will call
% buildpage'' and move them to the main vertical list. And only if
% this has happened the \aftergroup token gets executed.  Now given
% that we set \vsize to the largest possible dimen this means that all
% the remainder that was not used first time around will now end up on
% the main vertial list and only then \addendpenalty kicks in.

\output{%
\global\savedvsize\vsize
\global\vsize\maxdimen
%--- tracing --------------
\showthe\outputpenalty
\showlists
%--------------------------
\unvbox255
%
% above I claimed we put \outputpenalty back (which we should) but to
% make things more visible I put back a special penalty that can be
% easily recognised in \showlists
%
%  \penalty \ifnum\outputenalty=10000 0 \else \outputpenalty \fi
\penalty-777
\global\output{\ORtwo}%
}

% The macro \addendpenalty is used with \aftergroup from the output
% routine to gain control again. It adds a penalty to trigger the next
% output routine. However, we are quite likely in horizontal mode when
% the OR returns (just have seen the start of a paragraph) so we check
% for this. If true we remove the indentation box and end the
% paragraph. As a result all that get contributed is \parskip but no
% box (so \prevdepth is not touched). We signal with the penalty value
% whether or not we have seen hmode as we will have to remove that
% extra parskip in the next OR.

\ifhmode
\setbox0\lastbox\par\penalty-10010
\else
\penalty-10011
\fi}

% the second Or now should receive everything that was on the main
% vertical list with the recent contributions being empty (well empty
% except for a \penalty10000 that TeX puts in the place where it
% triggered the OR).
%
% What we have to do now is to remove the surplus \parskip at the
% bottom of 255 if we have been in hmode before. This is something we
% can determine by looking at the \outputpenalty that should be -10010
% in that case (otherwise -10011)
%
% then we save all of 255 in a spare box and return from the OR. To
% gain control afterwards we issue \aftergroup\XXX

\def\ORtwo{%
%--- tracing --------------
\showthe\outputpenalty
\showbox255\showlists
%--------------------------
\ifnum\outputpenalty=-10010
\setbox255=\vbox
{\unvbox255
\unskip  % this gets rid of the \parskip from hmode
}
\fi
\global\setbox\savedORbox\box255
\aftergroup\XXX
}

% Note that now the macro \XXX is immediately called when the OR ends
% as the recent contributions are empty now. Thus this macro now is
% getting us in a good shape:
%
% - it can access \prevdepth and \prevgraf (which is in fact having
% the same issue) and it can change them as necessary.
%
% - it has the complete main vertical list at its disposal (saved
% in \savedORbox)
%
%  - there is nothing left in recent contributions so anything
%  following is new material, so we can now arrange everything to our
%  liking and reprocess

\def\XXX{%
%--- tracing --------------
\showthe\prevdepth  % this is finally the outer one and we could
% change it if needed
\showlists          % nothing on it not even the penalty remains
% only prevdepth and prevgraf set (incorrectly)
%--------------------------
%  \global\vsize\savedvsize
\global\vsize20\baselineskip
\global\output{\plainoutput}%
\unvbox\savedORbox
}

% what we do above is set the vsize back to 20 baselines set up the
% plainoutput routine again and reprocess and we get 100% the same as
% in the initial test (well, in one place there is penalty 777 but
% that was just mark that spot, normally we would have \outputpenalty
% there which was 0. In the original there was nothing in this space
% only glue but that is equivalent

% and here is now the real test: we set a very short vsize so the
% first OR is triggered with \testmaterial and some part of it ends up
% in recent contributions.

\vsize=3\baselineskip

\testmaterial

\bye

• Even though i learned more TeX then I ever wanted to, I'm having trouble understanding all of this. I don't see why \setbox0=\vbox{\unvbox255} makes the next line too close, but before I care for that, I would like to know: what is a typical real world application where glue calculations fail as they do in the example? – Bananguin Sep 22 '14 at 12:39
• @user1129682 not sure you accept this as a "real world" example but latex-project.org/cgi-bin/ltxbugs2html?pr=tools/… is one of the reasons that I identified the issue. – Frank Mittelbach Sep 22 '14 at 13:18
• @user1129682 the \setbox simply throws away the material. It is a trivial way of making an example where the OR puts something back that has a different depth than what was calculated before. Real life examples would be, for example, to balance the material (producing a different depth) or inserting additional material that has a different depth (example in the OR chapter of the TeXbook) – Frank Mittelbach Sep 23 '14 at 9:17

When the OR returns something back to the main vertical list (RET material), then this RET material is inserted (rougly speaking, we don't consider inserts for this case) instead the material packed in box255. No matter if a nonempty remainder is or isn't after it. The interline glue between RET material and nonempty remainder is not calculated again by \baselineskip and \prevdepth because the skip is here already calculated. Only, what you need, is to emulate the depth of RET material is equal to the depth of the box255. On the other hand, if the remainder is empty, then the global \prevdepth keeps the value of box255 and the interline calculation assumes that the previous box was here with this depth. Only, what you need, is to emulate the depth of RET material is equal to the depth of the box255. So, there is no difference.

Because the OR knows both: the depth of box255 and the depth of RET material, it can do simply a correction. Note, I modified your example only in your special OR code.

\tracingonline=1

test with g to get a depth

\showthe\prevdepth % we see the prevdepth from the last line

\output{\edef\depthcclv{\the\dp255}%<<< --- this code was added by wipet
\setbox0=\vbox{\unvbox255}%
next line will be too close\par
\kern-\prevdepth \kern\depthcclv %<<< --- this line was added by wipet
\showlists
\showthe\prevdepth
}
\eject

% now we see that the prevdepth should be (and is 0pt) last line jiust contains
% characters without depth

\showlists
\showthe\prevdepth

% but now we got the old \prevdepth back even though it is no longer valid, as
% the OR simply pops the nest even though it is no longer valid.

% As a result we will get the wrong alignment on the next paragraph

Second line \par
\showlists

%  here we can see that  we are off by the 1.9...pt prevdepth as we should see
% 12pt baseline to baseline but  we don't

\output{\plainoutput}

\bigskip

But we really should see:

Next line will not be too close \par Second line

\bye

• Hi Petr, sorry but this is not solving the problem (though it appears to be on first glance). Try "test with \vrule depth 88pt to get a depth \lineskip=20pt" instead of the one with "g". As the line is thrown away we should never see \lineskip applied but it is now as the outer \prevdepth still believes it is 88pt. In other words, what your code is doing is to assume that the dp255 is somehow corresponding to the outer \prevdepth but it is not. – Frank Mittelbach Sep 16 '14 at 8:00
• Hi Frank. My attempt solves only \baslineskip grid because I supposed that only the line grid is needed. Of course, the full interline calculation with \lineskiplimit isn't possible even if you (theoretically) know the status of remainder (empty/nonempty) and you have the access to the global \prevdepth in OR. Because: if the remainder isn't empty and you have RET material in OR then there isn't possibility to do full calculation of interline glue between RET and remainder. You don't know the height of the first box of the remainder. And this calculation isn't repeated in main vertical list. – wipet Sep 16 '14 at 10:49
• The access to the global \prevdepth in OR cannot help you to solve the problem described in the previous comment. – wipet Sep 16 '14 at 10:50
• I guess you are right, it can not fully solve the issue, but it would at least allow you find out if you indeed have an issue. In other words, what is really needed for a full solution is to determine 100% whether or not recent contributions are empty. I finally found a way to actually make that determination (I think) just have to write it up and do a bit more testing. So far it looks as if it can resolve both questions. – Frank Mittelbach Sep 16 '14 at 11:59