Defining environments calling other environments (bug in 3.14159265-2.6-1.40.21)

Question

A strange error has suddenly appeared after I upgraded Debian, resulting in an update of texlive. Does anyone know the source of this error? How to fix it? I have used similar constructs for more than twenty years, and they have appeared safe and sound until now. I really don't want to rewire old class files if I can avoid it. My searches turn up false hits only. TIA

(EDIT) The original minimal example was solved and explained by David Carlisle below. The following example is less trivial.

% cat test.tex
\documentclass{article}
\newenvironment{testenv}{\def\foobar{1}\document}{The End\enddocument}
\begin{testenv}
Test\foorbar Test
\end{testenv}

Note, the use case is a class which defines several environments following the same pattern, and using \AtBeginDocument/\AtEndDocument is not straight forward.

On Debian buster, this works as expected

% pdflatex test.tex 
This is pdfTeX, Version 3.14159265-2.6-1.40.19 (TeX Live 2019/dev/Debian) (preloaded format=pdflatex)
 restricted \write18 enabled.
entering extended mode
(./test.tex
LaTeX2e <2018-12-01>
(/usr/share/texlive/texmf-dist/tex/latex/base/article.cls
Document Class: article 2018/09/03 v1.4i Standard LaTeX document class
(/usr/share/texlive/texmf-dist/tex/latex/base/size10.clo)) (./test.aux)
[1{/var/lib/texmf/fonts/map/pdftex/updmap/pdftex.map}] (./test.aux) )</usr/shar
e/texlive/texmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pfb>
Output written on test.pdf (1 page, 10790 bytes).
Transcript written on test.log.

On Debian bullseye, I get

pdflatex test.tex 
This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020/Debian) (preloaded format=pdflatex)
 restricted \write18 enabled.
entering extended mode
(./test.tex
LaTeX2e <2020-10-01> patch level 4
L3 programming layer <2021-01-09> xparse <2020-03-03>
(/usr/share/texlive/texmf-dist/tex/latex/base/article.cls
Document Class: article 2020/04/10 v1.4m Standard LaTeX document class
(/usr/share/texlive/texmf-dist/tex/latex/base/size10.clo))
(/usr/share/texlive/texmf-dist/tex/latex/l3backend/l3backend-pdftex.def)
(./test.aux)

! LaTeX Error: \begin{testenv} on input line 5 ended by \end{document}.

See the LaTeX manual or LaTeX Companion for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.7 \end{testenv}
                 
? 
[1{/var/lib/texmf/fonts/map/pdftex/updmap/pdftex.map}] (./test.aux) )
(\end occurred inside a group at level 1)

### semi simple group (level 1) entered at line 5 (\begingroup)
### bottom level</usr/share/texlive/texmf-dist/fonts/type1/public/amsfonts/cm/c
mr10.pfb>
Output written on test.pdf (1 page, 10794 bytes).
Transcript written on test.log.

If it matters, the results are the same with lualatex or latex.

Answer 1

This is unrelated to the version of TeX, or the operating system, it is a documented change in \document which had some adjustments to accomodate the new hook system.

Declaring the outer environment that way would be inefficient even in earlier releases where it did not give an error as it would place a group around the document causing tex to save all values set during the document to restore at the end of the group.

A version of the definition that works without error and without introducing a group level is

\documentclass{article}
\let\testenv\document
\let\endtestenv\enddocument
\begin{testenv}
Test
\end{testenv}

---

With the extended example added later you can insert the additional code using the hook mechanism

\documentclass{article}
\AddToHook{begindocument/before}{\def\foorbar{1}}
\AddToHook{enddocument}{The End}
\let\testenv\document
\let\endtestenv\enddocument
\begin{testenv}
Test\foorbar Test
\end{testenv}

although best practice would be to just use the hooks and not rename document so

\documentclass{article}
\AddToHook{begindocument/before}{\def\foorbar{1}}
\AddToHook{enddocument}{The End}

\begin{document}
Test\foorbar Test
\end{document}

Of course adding such code without different packages trying to re-define \document and tripping over each other is the reason for adding these standard hooks. In general it has proved a very compatible change at the document level, and does not affect most packages, although as you found, sorry, it does impact code that was previously patching in to \begin{document} "by hand".

See texdoc lthooks for more detail on the hook mechanism.

https://texdoc.org/serve/lthooks/0