Why is pgf 3.1 not compatible with tex?

Question

The release of pgf 3.1 on Jan the 6th broke a nightly build I'm running. Updating the package locally allowed me to reproduce the error, so I'm certain that's what caused it.

Here is a broken MWE:

\input tikz
Help?
\bye

Here is the log of running tex on it:

This is TeX, Version 3.14159265 (TeX Live 2018) (preloaded format=tex)
(./main.tex
(/usr/local/texlive/2018/texmf-dist/tex/plain/pgf/frontendlayer/tikz.tex
(/usr/local/texlive/2018/texmf-dist/tex/plain/pgf/basiclayer/pgf.tex
(/usr/local/texlive/2018/texmf-dist/tex/plain/pgf/utilities/pgfrcs.tex
(/usr/local/texlive/2018/texmf-dist/tex/generic/pgf/utilities/pgfutil-common.te
x
(/usr/local/texlive/2018/texmf-dist/tex/generic/pgf/utilities/pgfutil-common-li
sts.tex))
(/usr/local/texlive/2018/texmf-dist/tex/generic/pgf/utilities/pgfutil-plain.def
(/usr/local/texlive/2018/texmf-dist/tex/generic/oberdiek/atbegshi.sty
(/usr/local/texlive/2018/texmf-dist/tex/generic/oberdiek/infwarerr.sty)
(/usr/local/texlive/2018/texmf-dist/tex/generic/oberdiek/ltxcmds.sty)
(/usr/local/texlive/2018/texmf-dist/tex/generic/oberdiek/ifpdf.sty)))
(/usr/local/texlive/2018/texmf-dist/tex/generic/pgf/utilities/pgfrcs.code.tex
(/usr/local/texlive/2018/texmf-dist/tex/generic/pgf/pgf.revision.tex)))
(/usr/local/texlive/2018/texmf-dist/tex/plain/pgf/basiclayer/pgfcore.tex
(/usr/local/texlive/2018/texmf-dist/tex/plain/pgf/systemlayer/pgfsys.tex
(/usr/local/texlive/2018/texmf-dist/tex/plain/pgf/utilities/pgfrcs.tex)
(/usr/local/texlive/2018/texmf-dist/tex/generic/pgf/systemlayer/pgfsys.code.tex
(/usr/local/texlive/2018/texmf-dist/tex/generic/pgf/utilities/pgfkeys.code.tex
(/usr/local/texlive/2018/texmf-dist/tex/generic/pgf/utilities/pgfkeysfiltered.c
ode.tex))
! Undefined control sequence.
\pgfkeyssetevalue ...gfkeys@temptoks =\scantokens
                                                  \expandafter {\expandafter...

\pgfkeys@ifcsname ...\fi \ifpgfkeys@csname@test #2
                                                  \else #3\fi
\pgfkeys@ifcsname ...gfkeys@csname@test #2\else #3
                                                  \fi
\pgfkeys@ifcsname ...gfkeys@csname@test #2\else #3
                                                  \fi
\pgfkeys@unpack ...pgfeov \else \pgfkeys@case@one
                                                  \fi \fi
\pgfkeys@@normal ...pgfkeysnovalue =\pgfkeys@stop
                                                  \pgfkeys@parse
...
l.17 \pgfkeys{/pgf/.is family}

The MWE breaks using tex, ptex and uptex. It works using etex, pdftex, xetex, luatex, eptex and euptex.

It seems to break when using anything that's missing etex extensions. That makes sense since \scantokens is such an extension.

I'm a bit surprised by the breaking change, and I might simply be missing something.

Does this actually mean that pgf lost some compatibility? If that's the case, does anyone know the rationale behind the decision (and perhaps a link to the commit)?

NOTE: There's no complaint here, I'd just like to learn more. I know infinite retrocompatibility is never to be expected, and I think the maintainers of pgf/TikZ are amazing.

Answer 1

The answer why PGF 3.1 does not support Knuth TeX is two-fold.

1. It is the current year. By now e-TeX is over 20 years old and has simplified TeX development tremendously. I consider it a bug on your side that you are not using e-TeX. Unfortunately TeX Live ships a tex binary which does not enable e-TeX extensions. You have to use etex instead.

2. Development of TikZ/PGF has been dead for a couple of years because Till Tantau has apparently abandoned the project entirely and Christian Feuersänger has been busy in offline life. Around Christmas 2018 I was contacted by Christian (via Stefan Pinnow) to join the PGF development team so that bugs can be fixed and a new release can be prepared. Christian was keen to publish the release around Christmas time before getting back to work. This left very little time for testing and a lot of bugs and broken bug fixes ended up as part of the 3.1 release. My goal is to actively participate in TeX Live 2019 pretest to eradicate all the bugs before the next release. Bottomlined I apologize for this poor quality release but the circumstances forced us to release quickly.

On another note, I'd like to ask everyone to please report bugs on the official PGF bugtracker rather than on Stack Exchange. It's unfortunate that you don't earn internet points there, but it simplifies the work of the developers quite a lot. If you really want to indulge those sweet reputation points, you can post a question on Stack Exchange in addition to your bug report on the official PGF bugtracker. Thanks.

If you are unsure whether what you are observing is a bug and you have already posted it here, but were told to open a bug report, please repeat all the necessary information to reproduce the problem on the PGF bugtracker as well. Bug reports which only contain a link are not very nice as they have an undertone that you do not value the developers' time.