Is there a way to get at the value of \prevdepth from the main vertical list inside the output routine?

Question

To motivate that (admittedly) strange question:

When TeX builds a vertical list it keeps track of the depth of the last box added in a special variable called \prevdepth which is used to calculate baselineskip glue for paragraph lines. When a new vertical list is being built (and the OR starts a new vertical list) TeX reinitializes \prevdepth and when processing ends it resets it to the old (outer) value.

With normal output routines (that ship out pages) that's okay as the \prevdepth (after the output routine ends) will match the last box on the main vertical list if there is a remainder that was not put into box255. If the main vertical list is empty the value is wrong but that doesn't make a difference because TeX will use \topskip on top of the first line so it is essentially ignored.

But the situation is different if the output routine puts material back to be reprocessed. In that case the last box on the main vertical list is a) either the box that was not used in the OR and that has a depth matching \prevdepth or it is the last box from the stuff that the OR contributed (which may have a different depth).

So to make baselineskip calculations work correctly we would need to know which situation we are in: has everything been packaged into box255 or is there an unused remainder? If that question could be answered that would also solve the issue but this is to my knowledge not possible.

An alternative solution would be to pretend that the material generated by the OR has the same depth as the outer \prevdepth but for this I would somehow need to know its value. Knowing its value is simple if we force the OR with an explicit penalty because then I can just save it and used the saved value. But what can be done if the OR is called by TeX on its own?

That's really the question above ... any idea (as sneaky as it needs to be) to somehow get at this value in this particular case?

Or alternatively, any method to clearly determine whether or not the main vertical list is emptied.

Please note that I'm looking for a solution that works with all TeX engines, so doesn't make use of, say, lua programming.

To motivate a bit further, here is a short plain TeX file that shows the general problem (though here it could be fixed as we explicitly force the output routine):

\tracingonline=1
\showboxbreadth\maxdimen\showboxdepth\maxdimen

test with g to get a depth

\showthe\prevdepth % we see the prevdepth from the last line

% now assume we have some OR that traps the data and does something with it
% afterwards it is pushing back new or changed data t build pages, In the example I
% simply dropped the collected data and replaced it with soemthing else which has a different 
% \prevdepth. A real life example would be rebalancing existing material.

\output{\setbox0=\vbox{\unvbox255}%
  next line will be too close\par
  \showlists
  \showthe\prevdepth
  }
  \eject

  % now we see that the prevdepth should be (and is 0pt) last line jiust contains
  % characters without depth

  \showlists
  \showthe\prevdepth

  % but now we got the old \prevdepth back even though it is no longer valid, as
  % the OR simply pops the nest even though it is no longer valid.

  % As a result we will get the wrong alignment on the next paragraph

  Second line \par
    \showlists

   %  here we can see that  we are off by the 1.9...pt prevdepth as we should see
   % 12pt baseline to baseline but  we don't

  \output{\plainoutput}

  \bigskip

  But we really should see:

  Next line will not be too close \par Second line

    \bye

If we run this we get:

and if we look into the log we can also see how the wrong \prevdepth is messing up vertical spacing:

> 1.94444pt.
l.6 \showthe\prevdepth
                       % we see the prevdepth from the last line
? 

### internal vertical mode entered at line 18 (\output routine)
\hbox(6.94444+0.0)x469.75499, glue set 337.7548fil
.\hbox(0.0+0.0)x20.0
.\tenrm n
.\tenrm e
.\tenrm x
.\tenrm t
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm l
.\tenrm i
.\tenrm n
.\tenrm e
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm w
.\tenrm i
.\tenrm l
.\tenrm l
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm b
.\kern0.27779
.\tenrm e
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm t
.\tenrm o
.\kern0.27779
.\tenrm o
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm c
.\tenrm l
.\tenrm o
.\tenrm s
.\tenrm e
.\penalty 10000
.\glue(\parfillskip) 0.0 plus 1.0fil
.\glue(\rightskip) 0.0
prevdepth 0.0, prevgraf 1 line
### vertical mode entered at line 0
### recent contributions:
\penalty 10000
prevdepth 1.94444, prevgraf 1 line

! OK.
<output> ...line will be too close\par \showlists 
                                                  \showthe \prevdepth }
\break ->\penalty -\@M 

l.18   \eject

? 
> 0.0pt.
<output> ...se\par \showlists \showthe \prevdepth 
                                                  }
\break ->\penalty -\@M 

l.18   \eject

? 

### vertical mode entered at line 0
### current page:
\glue(\topskip) 3.05556
\hbox(6.94444+0.0)x469.75499, glue set 337.7548fil
.\hbox(0.0+0.0)x20.0
.\tenrm n
.\tenrm e
.\tenrm x
.\tenrm t
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm l
.\tenrm i
.\tenrm n
.\tenrm e
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm w
.\tenrm i
.\tenrm l
.\tenrm l
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm b
.\kern0.27779
.\tenrm e
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm t
.\tenrm o
.\kern0.27779
.\tenrm o
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm c
.\tenrm l
.\tenrm o
.\tenrm s
.\tenrm e
.\penalty 10000
.\glue(\parfillskip) 0.0 plus 1.0fil
.\glue(\rightskip) 0.0
\penalty 10000
total height 10.0
 goal height 643.20255
prevdepth 1.94444, prevgraf 1 line

! OK.
l.23   \showlists

? 

> 1.94444pt.
l.24   \showthe\prevdepth

? 
### vertical mode entered at line 0
### current page:
\glue(\topskip) 3.05556
\hbox(6.94444+0.0)x469.75499, glue set 337.7548fil
.\hbox(0.0+0.0)x20.0
.\tenrm n
.\tenrm e
.\tenrm x
.\tenrm t
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm l
.\tenrm i
.\tenrm n
.\tenrm e
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm w
.\tenrm i
.\tenrm l
.\tenrm l
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm b
.\kern0.27779
.\tenrm e
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm t
.\tenrm o
.\kern0.27779
.\tenrm o
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm c
.\tenrm l
.\tenrm o
.\tenrm s
.\tenrm e
.\penalty 10000
.\glue(\parfillskip) 0.0 plus 1.0fil
.\glue(\rightskip) 0.0
\penalty 10000
\glue(\parskip) 0.0 plus 1.0
\glue(\baselineskip) 3.11111
\hbox(6.94444+0.0)x469.75499, glue set 400.31046fil
.\hbox(0.0+0.0)x20.0
.\tenrm S
.\tenrm e
.\tenrm c
.\tenrm o
.\tenrm n
.\tenrm d
.\glue 3.33333 plus 1.66666 minus 1.11111
.\tenrm l
.\tenrm i
.\tenrm n
.\tenrm e
.\penalty 10000
.\glue(\parfillskip) 0.0 plus 1.0fil
.\glue(\rightskip) 0.0
total height 20.05556 plus 1.0
 goal height 643.20255
prevdepth 0.0, prevgraf 1 line

! OK.
l.32     \showlists

Answer 1

@wipet in his answer has shown how to resolve the problem as long as we can assume that that the \prevdepth of all material in the document is sufficiently small (that it lies below \maxdepth in fact). In that case we can use the depth of box 255 as a measure for the \prevdepthcalculations that will have taken place if there is any remainder in recent contributions, and use this to adjust the new depth to match that. An if there isn't any remainder then this doesn't really matter either. This trick is in fact already mentioned by Don Knuth in the TeXbook where he discusses an output routine that add index headings in random places in the text (and normal text has this nice property of having a depth smaller than \maxdepth ... normally but unfortunately not always.

However, this stops working if this can't guaranteed and the general case is what I was and am after. He is also correct in stating that finding out what the value of \prevdepth on the main vertical list is isn't really going to help us as such, so my question above was partly incorrect: we also need to know if there is in fact something on the recent contributions and if so deal with it.

So what I tried to get my head around over the last days if that is really impossible to find out within base TeX (or if it needs an extension in luatex or is already available there and resolved in ConTeXt ... would be interesting to learn about it beside that claim @Martin you made). And as it stands now it is possible in TeX after all. The solution is a bit complex and perhaps it can be simplified further but it is not so complex that it can't be used (and hopefully not too complex that I overlooked some cases).

The main idea is not to try to determine the situation directly but instead use 2 output routines in succession that just ensure that the recent contributions are empty. Along the way we gather enough information to afterwards dissect the material gathered and do whatever we like.

My initial idea of using \aftergroup to regain control and have a look doesn't work because the token inserted that way will not be executed when the output routine ends. Instead TeX immediately calls again on a procedure "buildpage" that takes anything on recent contributions and moves it to the main vertical list and only once that has happened my inserted token is looked at (in other words too late).

So the more complicated approach is that the first OR puts box255 back but only it has changed \vsize to the largest possible dimen. Additionally it uses \aftergroup so that we gain control later again. As we have changed \vsize we will get everything including the material recent contributions onto the main vertical list and only then our control token kicks in. Finally we change the output routine to a second one and then return.

The token inserted by \aftergroup will then issue a forcing penalty (actually it does a little more, see below) so that everything is grabbed and the second OR is called.

Inside that OR we are now in a better situation:

• we know that recent contributions is empty (except for a (\penalty 10000)
• we can store away what was gathered in box255
• or we could manipulate it using \vsplit etc to get, for example the amount split of that we would have gotten in the first OR
• and we can use \aftergroup to gain control after the OR has ended.
• the latter allows us to change \prevdepth on the main vertical list to represent whatever is needed

And that should (I believe) do the trick completely.

Here is the (more or less documented code including a bit of test data to play out various scenarios). It is a bit longish but largely because I tried to properly document the most important aspects and some of the more subtle points. Enjoy:

\tracingoutput1
\showboxbreadth\maxdimen\showboxdepth\maxdimen
\tracingpages1
\tracingonline1

\vsize20\baselineskip

\lineskip=13pt  % for identifications

% this is our trial material used below. We will arrange things so
% that the first para will be longer than a page so that we will end
% up with some material on recent contributions. The OR is actually
% then triggered when the first ``p'' is seen. Alternatively one can
% uncomment the \vskip or the \penalty in which case the OR will be
% triggered by them or you could uncomment ``Line2 and3'' then the
% break happens somewhere in the middle of the first paragraph (in
% vmode inn that case)

\def\testmaterial{%
  Line 1 \hfil\break
%  Line 2 \hfil\break
%  Line 3 \hfil\break
  \vadjust{\penalty -333 }
  some text some text some text some text some text some text
  some text some text some text
   and some more text ggg \vrule depth 88pt

%\ vskip 17pt
% \penalty 15

  pppppppppppppppppppppppppp

  \showlists
}


% now this here is just to see the whole stuff being processed by the
% normal OR and see the \showlists result for it. One can then compare
% that to the showlists result we get later to check for differences

\testmaterial

\vfill

\eject


%==============================================

\newdimen\savedvsize
\newbox\savedORbox


% now for a set of special output routines:
% the first one does the following
%
% - save away current \vsize and set it to \maxdimen
% - then unbox 255 and readd the output penalty (unless it is 10000)
% - set up a new output routine for the next time
% - finally install control with \aftergroup\addendpenalty
%

% The point here is that the \aftergroup token is not actually
% directly executed the moment the OR ends. If TeX ends an OR it looks
% at the recent contribution and if they are not empty it will call
% ``buildpage'' and move them to the main vertical list. And only if
% this has happened the \aftergroup token gets executed.  Now given
% that we set \vsize to the largest possible dimen this means that all
% the remainder that was not used first time around will now end up on
% the main vertial list and only then \addendpenalty kicks in.


\output{%
  \global\savedvsize\vsize
  \global\vsize\maxdimen
%--- tracing --------------
  \showthe\outputpenalty
  \showlists
%--------------------------
  \unvbox255
%
% above I claimed we put \outputpenalty back (which we should) but to
% make things more visible I put back a special penalty that can be
% easily recognised in \showlists
%
%  \penalty \ifnum\outputenalty=10000 0 \else \outputpenalty \fi
  \penalty-777
  \global\output{\ORtwo}%
  \aftergroup\addendpenalty
}

% The macro \addendpenalty is used with \aftergroup from the output
% routine to gain control again. It adds a penalty to trigger the next
% output routine. However, we are quite likely in horizontal mode when
% the OR returns (just have seen the start of a paragraph) so we check
% for this. If true we remove the indentation box and end the
% paragraph. As a result all that get contributed is \parskip but no
% box (so \prevdepth is not touched). We signal with the penalty value
% whether or not we have seen hmode as we will have to remove that
% extra parskip in the next OR.

\def\addendpenalty{%
 \ifhmode
   \setbox0\lastbox\par\penalty-10010
 \else
   \penalty-10011 
 \fi}


% the second Or now should receive everything that was on the main
% vertical list with the recent contributions being empty (well empty
% except for a \penalty10000 that TeX puts in the place where it
% triggered the OR).
%
% What we have to do now is to remove the surplus \parskip at the
% bottom of 255 if we have been in hmode before. This is something we
% can determine by looking at the \outputpenalty that should be -10010
% in that case (otherwise -10011)
%
% then we save all of 255 in a spare box and return from the OR. To
% gain control afterwards we issue \aftergroup\XXX

\def\ORtwo{%
%--- tracing --------------
  \showthe\outputpenalty
  \showbox255\showlists
%--------------------------
  \ifnum\outputpenalty=-10010
    \setbox255=\vbox
      {\unvbox255
       \unskip  % this gets rid of the \parskip from hmode
      }
  \fi
  \global\setbox\savedORbox\box255
  \aftergroup\XXX
}

% Note that now the macro \XXX is immediately called when the OR ends
% as the recent contributions are empty now. Thus this macro now is
% getting us in a good shape:
%
% - it can access \prevdepth and \prevgraf (which is in fact having
% the same issue) and it can change them as necessary.
%
% - it has the complete main vertical list at its disposal (saved
% in \savedORbox)
%
%  - there is nothing left in recent contributions so anything
%  following is new material, so we can now arrange everything to our
%  liking and reprocess


\def\XXX{%
%--- tracing --------------
  \showthe\prevdepth  % this is finally the outer one and we could
                      % change it if needed
  \showlists          % nothing on it not even the penalty remains
                      % only prevdepth and prevgraf set (incorrectly)
%--------------------------
%  \global\vsize\savedvsize
  \global\vsize20\baselineskip
  \global\output{\plainoutput}%
  \unvbox\savedORbox
}

% what we do above is set the vsize back to 20 baselines set up the
% plainoutput routine again and reprocess and we get 100% the same as
% in the initial test (well, in one place there is penalty 777 but
% that was just mark that spot, normally we would have \outputpenalty
% there which was 0. In the original there was nothing in this space
% only glue but that is equivalent


% and here is now the real test: we set a very short vsize so the
% first OR is triggered with \testmaterial and some part of it ends up
% in recent contributions.

\vsize=3\baselineskip

\testmaterial


\bye

Answer 2

When the OR returns something back to the main vertical list (RET material), then this RET material is inserted (rougly speaking, we don't consider inserts for this case) instead the material packed in box255. No matter if a nonempty remainder is or isn't after it. The interline glue between RET material and nonempty remainder is not calculated again by \baselineskip and \prevdepth because the skip is here already calculated. Only, what you need, is to emulate the depth of RET material is equal to the depth of the box255. On the other hand, if the remainder is empty, then the global \prevdepth keeps the value of box255 and the interline calculation assumes that the previous box was here with this depth. Only, what you need, is to emulate the depth of RET material is equal to the depth of the box255. So, there is no difference.

Because the OR knows both: the depth of box255 and the depth of RET material, it can do simply a correction. Note, I modified your example only in your special OR code.

\tracingonline=1
\showboxbreadth\maxdimen\showboxdepth\maxdimen

test with g to get a depth

\showthe\prevdepth % we see the prevdepth from the last line

\output{\edef\depthcclv{\the\dp255}%<<< --- this code was added by wipet
  \setbox0=\vbox{\unvbox255}%
  next line will be too close\par
  \kern-\prevdepth \kern\depthcclv %<<< --- this line was added by wipet
  \showlists
  \showthe\prevdepth
  }
  \eject

  % now we see that the prevdepth should be (and is 0pt) last line jiust contains
  % characters without depth

  \showlists
  \showthe\prevdepth

  % but now we got the old \prevdepth back even though it is no longer valid, as
  % the OR simply pops the nest even though it is no longer valid.

  % As a result we will get the wrong alignment on the next paragraph

  Second line \par
    \showlists

   %  here we can see that  we are off by the 1.9...pt prevdepth as we should see
   % 12pt baseline to baseline but  we don't

  \output{\plainoutput}

  \bigskip

  But we really should see:

  Next line will not be too close \par Second line

    \bye