pgfpositionnodelater with nested \tikz environments (Issue with PGF 2.1 -> 3.0 upgrade)

Question

I'm trying to use \pgfpositionnodelater to create and measure nodes before placing them. (Their size influences where they should be placed)

This works fine until I try to use \pgfpositionnode later with a node that has a \tikz[]{..} environment in its body.

I'm using nested tikz environments because I have a number of complex macros that need to be nested very deeply, and trying to do this manually leads to an absurd amount of code for a diagram that could be specified much more cleanly.

I've got a minimum working example that tries to generate a diagram like this:

I want a set of macros that could be nested arbitrarily to create a tree like structure, where every parent box is tall enough to cover all its children. (Making this particular diagram without the features I use would be easy, However I'm trying to cover the general case.)

The Code that should generate the image, give or take some visible debugging bounding boxes and incorrectly set inner sep values:

\documentclass[crop]{article}% 'crop' is the default for v1.0, before it was 'preview'

\usepackage{tikz}
 \usepackage{xparse}
 \usepackage{ifthen}

 \usetikzlibrary{
   positioning,
 }

 \makeatletter
 \ExplSyntaxOn

 % Define an identifier and a set of TIKZ commands so that you can get bounding
 % box information before you place a node. 
 % #1 = identifier for the stuff you place 
 % #2 = tikz code you wish to place later
 \DeclareDocumentCommand \placenodelater {d() +m}{
   % Debug
   \typeout{Starting_placenodelater_for_#1.}
   % The macro the \pgfpositionnodelateruses as generated by the identifier 
   % given. 
   \cs_new_protected:cpn {pgf@#1-save-macro} {
     % calculate width and height
     \pgfmathsetmacro{\pgf@tempwid}{
       \pgfpositionnodelatermaxx - \pgfpositionnodelaterminx
     }
     \pgfmathsetmacro{\pgf@temphei}{
       \pgfpositionnodelatermaxy - \pgfpositionnodelaterminy
     }
     % Save all the location registers 
     \cs_gset_eq:cN   {pgf@#1-name}     \pgfpositionnodelatername
     \cs_gset_eq:cN   {pgf@#1-minx}     \pgfpositionnodelaterminx
     \cs_gset_eq:cN   {pgf@#1-miny}     \pgfpositionnodelaterminy
     \cs_gset_eq:cN   {pgf@#1-maxx}     \pgfpositionnodelatermaxx
     \cs_gset_eq:cN   {pgf@#1-maxy}     \pgfpositionnodelatermaxy
     \cs_gset_eq:cN   {pgf@#1-width}    \pgf@tempwid
     \cs_gset_eq:cN   {pgf@#1-height}   \pgf@temphei
     \box_if_exist:cF {pgf@#1-box}      {\box_new:c {pgf@#1-box}}
     \box_gset_eq:cN  {pgf@#1-box}      \pgfpositionnodelaterbox
     % Debug information
     \typeout{__pgf@#1-save-macro_data_:}
     \typeout{_____name___:_\use:c{pgf@#1-name}}
     \typeout{_____min-x__:_\use:c{pgf@#1-minx}}
     \typeout{_____max-x__:_\use:c{pgf@#1-maxx}}
     \typeout{_____min-y__:_\use:c{pgf@#1-miny}}
     \typeout{_____max-y__:_\use:c{pgf@#1-maxy}}
     \typeout{_____height_:_\use:c{pgf@#1-height}}
     \typeout{_____width__:_\use:c{pgf@#1-width}}
   }

   {
     \exp_args:Nc \pgfpositionnodelater {pgf@#1-save-macro} #2
   }
   % Debug
   \typeout{Ending_placenodelater_for_#1.}
 }

 % Place the node previously specified with the \placenodelater command
 % #1 = identifier for nodes already saved by \placenodelater
 % #2 = location for the node to be placed at
 \DeclareDocumentCommand \placenodenow {d() +m}{
   % Debug
   \typeout{Starting_placenodenow_for_#1.}
   % Move the various variables back
   \cs_set_eq:Nc  \pgfpositionnodelatername {pgf@#1-name}
   \cs_set_eq:Nc  \pgfpositionnodelaterminx {pgf@#1-minx}
   \cs_set_eq:Nc  \pgfpositionnodelaterminy {pgf@#1-miny}
   \cs_set_eq:Nc  \pgfpositionnodelatermaxx {pgf@#1-maxx}
   \cs_set_eq:Nc  \pgfpositionnodelatermaxy {pgf@#1-maxy}
   \box_set_eq:Nc \pgfpositionnodelaterbox  {pgf@#1-box}
   % Call into PGF to place the node
   \pgfpositionnodenow{#2}
   % Debug information at time of placement
   \typeout{__#1_data_:}
   \typeout{_____name___:_\use:c{pgf@#1-name}}
   \typeout{_____min-x__:_\use:c{pgf@#1-minx}}
   \typeout{_____max-x__:_\use:c{pgf@#1-maxx}}
   \typeout{_____min-y__:_\use:c{pgf@#1-miny}}
   \typeout{_____max-y__:_\use:c{pgf@#1-maxy}}
   \typeout{_____height_:_\use:c{pgf@#1-height}}
   \typeout{_____width__:_\use:c{pgf@#1-width}}
   % Debug
   \typeout{Ending_placenodenow_for_#1.}
 }

 % #1 = Module Name 
 % #2 = Module Label
 % #3 = Submodule definitions
 \DeclareDocumentCommand \nestedmodule {d() +m +m} {
   % If there was a module before us, we're right below it
   % otherwise anywhere works
 %  \cs_if_exist_use:cTF {prevnode}
 %  {
 %    \tikzstyle{modulepositioning}=[
 %      ,anchor=north~west
 %      ,below=of~\use:c{prevnode}.south~west
 %    ]
 %  }
 %  {
 %    \tikzstyle{modulepositioning}=[]
 %  }

   \ifthenelse{\isundefined\prevnode}
   {
     \tikzstyle{modulepositioning}=[]
   }
   {
     \tikzstyle{modulepositioning}=[
       ,below=of~\prevnode.south~west
       ,anchor=north~west
     ]
   }
   % Create the enclosing node for our module
   \node[
       ,rectangle
       ,modulepositioning
       ,draw,red
   ](#1){
     \tikz[remember~picture]{
       % Create all the submodules before we place them
       \placenodelater(#1-mod-store){ 
         % Create node containing submodules
         \node[
           ,draw,blue
           ,anchor=west
           ,remember~picture
         ](#1modbox){
           \tikz[remember~picture]{
               \let\prevnode\undefined
               % Place all subnodes 
               #3
           }
         };
       }
       \pgfmathsetmacro{\centralheight}{\use:c{pgf@#1-mod-store-height}}
       % Create a main node that's large
         \node[
         ,draw,black
         ,remember~picture
         ,inner~sep=5pt
         ,minimum~width=40pt
         ,minimum~height=\centralheight
       ](#1label){
         #2
       };
 %      % Place sub-modules
       \placenodenow(#1-mod-store){\pgfpointanchor{#1label}{east}}
     }
   };
   % Set self as previous for next node
 %  \cs_set_eq:cN {prevnode} {#1}
     \def\prevnode{#1}
 } 

 \ExplSyntaxOff
 \makeatother
\begin{document}
 \begin{tikzpicture}[remember picture,inner sep =1pt,node distance = 5pt]
     \nestedmodule(root){root}{
       \nestedmodule(port-1){p1}{
         \nestedmodule(port-1-1){p11}{
         };
         \nestedmodule(port-1-2){p12}{
         };
       };
       \nestedmodule(port-2){p2}{
       };
     };
 \end{tikzpicture}
\end{document} 

Playing with the code a little shows that using \placenodelater with a normal node works fine, but the \tikz environment in that node isn't displayed. (Other sections of the label work, as do parallel elements.

Edit: I've managed to get a setup with PGF-2.1 working, and the above example (it changed with the edit)

The PGF-2.1 (and correct) output is:

The Broken output with PGF-3.0 is :

This is still a problem since I need tools that only work with PGF-3.0 in other parts of my document. I'm also really curious what difference (or set of difference) is causing the change.

---

I also have a full .sty file for the type of diagram that I am actually trying to make, that used this technique to successfully generate diagrams.

However I updated TeXLive and it broke. I'm honestly not sure how to fix this, or even debug the issue, I think I've reduced it to the issue the above MWE shows and that I ask the question about, but I'm not 100% sure of that.

(A diagram and the .sty file can be found here, these did work before I updated, even if I only created the MWE after the update. It tries to generate an image vaguely like one of the blocks in this image, Along with a random additional node, and a block that wraps those. I would post what it's actually supposed to create but I didn't save an image of the output before the update broke it.)

Edit : I've just rendered versions of this document's image with PGF-2.1 (approximately correct) and with PGF-3.0 (completely wrong).

---

Edit, May 13th:

Okay, this is driving me insane.

Further floundering has resulted in the following code:

% All earlier code is the same as before

% #1 = Module Name 
% #2 = Module Label
% #3 = Submodule definitions
\DeclareDocumentCommand \nestedmodule {r() +m +m} {
  % If there was a module before us, we're right below it
  % otherwise anywhere works
%  \cs_if_exist_use:cTF {prevnode}
%  {
%    \tikzstyle{modulepositioning}=[
%      ,anchor=north~west
%      ,below=of~\use:c{prevnode}.south~west
%    ]
%  }
%  {
%    \tikzstyle{modulepositioning}=[]
%  }

  \ifthenelse{\isundefined\prevnode}
  {
    \tikzstyle{modulepositioning}=[]
  }
  {
    \tikzstyle{modulepositioning}=[
      ,below=of~\prevnode.south~west
      ,anchor=north~west
    ]
  }
  % Create the enclosing node for our module
  \node[
      ,rectangle
      ,modulepositioning
      ,draw,red
  ](#1){
    \tikz[remember~picture]{
      % Create all the submodules before we place them
      \placenodelater(#1-mod-store){ 
        % Create node containing submodules
        \node[
          ,draw,blue
          ,anchor=west
          ,remember~picture
        ](#1modbox){
          \tikz[remember~picture]{
            \placenodelater(#1-welp){  % Make sure no prvious node is defined
              \let\prevnode\undefined
              % Place all subnodes 
              \node[draw,orange]{test};
              #3
            }
            \placenodenow(#1-welp){\pgfpointorigin}
          }
        };
      }
      \pgfmathsetmacro{\centralheight}{\use:c{pgf@#1-mod-store-height}}
      % Create a main node that's large
      \placenodelater(#1-more-welp){
        \node[
        ,draw,green
        ,remember~picture
        ,minimum~height=\centralheight
      ](#1label){
        #2
      };
    }
    \placenodenow(#1-more-welp){\pgfpointorigin}
%      % Place sub-modules
      \placenodenow(#1-mod-store){\pgfpointanchor{#1label}{east}}
    }
  };
  % Set self as previous for next node
%  \cs_set_eq:cN {prevnode} {#1}
    \def\prevnode{#1}
} 

\ExplSyntaxOff
\makeatother
\begin{document}
\begin{tikzpicture}[remember picture]
    \nestedmodule(root){root}{
      \nestedmodule(port-1){p1}{
        \nestedmodule(port-1-1){p11}{
        };
        \nestedmodule(port-1-2){p12}{
        };
      };
      %\nestedmodule(port-2){p2}{
      %};
    };
\end{tikzpicture}
\end{document} 

With the following result:

What I did was take every nested instance of a \tikz environment and stick another \placenodelater,\placenodenow pair in there.

This somewhat works, but it makes the actual issue more obvious. Namely, \pgfpositionnodelater will only save the last node in the scope it's called it, instead of taking all of them and putting a bounding box around it.

When I was using a nested \tikz environment I was placing the nodes in that nested environment, and then the external container node. Because only the last one is placed, only it is rendered.

The only solution that comes to mind is using \positionnodelater for every internal node and manually replicating the effects of the positioning library. This would be incredibly awkward, and would require me to override most of the core tikz constructs, so I'm still looking for a way to solve this.

---

Edit: I've posted a workaround as an answer, but I would still appreciate a genuine solution to the problem. As it is there's too many hacky things going on at any point for it to be robust, or easy to use.

Answer 1

So I've gone ahead and developed and absolutely awful workaround.

Namely placing the various nodes on an invisible layer to get their height and width, and then placing them again on the actual canvas.

It is worth noting that if you're using recursion in your macros, this will take exponential time relative to the depth of your recursion.

I'm posting it here for posterity, but let me state for the record that it is an awful way to solve this problem, and I've convinced myself that how PGF 3.0 handles recursion is a bug.

\documentclass[crop]{article}

 \usepackage{tikz}
 \usepackage{xparse}
 \usepackage{ifthen}

 \usetikzlibrary{
   positioning,
 }

 \makeatletter
 \ExplSyntaxOn

% Width and height detection from : http://tex.stackexchange.com/questions/8691/widthof-within-tikzpicture
\newcommand\getwidthofnode[2]{%
    \pgfextractx{\pgf@xb}{\pgfpointanchor{#2}{east}}%
    \pgfextractx{\pgf@xa}{\pgfpointanchor{#2}{west}}%
    \pgfmathsetmacro{#1}{\pgf@xb-\pgf@xa}%
}

\newcommand\getheightofnode[2]{%
    \pgfextracty{\pgf@yb}{\pgfpointanchor{#2}{north}}%
    \pgfextracty{\pgf@ya}{\pgfpointanchor{#2}{south}}%
    \pgfmathsetmacro{#1}{\pgf@yb -\pgf@ya}%
}

% Invisible layer code from : http://tex.stackexchange.com/questions/237109/how-to-have-an-invisible-layer-in-tikz
\pgfdeclarelayer{pgf@invisible}
\pgfsetlayers{main,pgf@invisible}
\def\pgfsetinvisiblelayers#1{\def\pgf@layers@invisible{#1}}
\pgfsetinvisiblelayers{pgf@invisible}

\def\pgf@dolayer#1,#2,\relax{%
  % Is layer declared as invisible...?
  \edef\pgf@marshal{\noexpand\pgfutil@in@{,#1,}{,\pgf@layers@invisible,}}%
  \pgf@marshal%
  \ifpgfutil@in@% Yep. So do nothing.
  \else% Nope. So, insert box.
    \def\pgf@test{#1}%
    \ifx\pgf@test\pgf@maintext%
      \box\pgf@layerbox@main%
    \else%
      \pgfsys@beginscope%
        \expandafter\box\csname pgf@layerbox@#1\endcsname%
      \pgfsys@endscope%
    \fi%
  \fi
  \def\pgf@test{#2}%
  \ifx\pgf@test\pgfutil@empty%
  \else%
    \pgf@dolayer#2,\relax%
  \fi%
}

 % Define an identifier and a set of TIKZ commands so that you can get bounding
 % box information before you place a node. 
 % #1 = identifier for the stuff you place 
 % #2 = tikz code you wish to place later
 \DeclareDocumentCommand \placenodelater {d() +m}{
   % Debug
   \typeout{Starting_placenodelater_for_#1.}    

   % Save the Nodes to be placed as a macro
   \cs_set_protected:cpn {pgf@#1-data}{#2}

   % Place them on an invisible layer
   \begin{pgfonlayer}{pgf@invisible}
     \node[
       ,inner~sep=0
       ,outer~sep=0
     ](pgf@#1-invisible-placement){
       \tikz[]{
         \use:c {pgf@#1-data}
       }
     };
   \end{pgfonlayer}

  % Get the height and width (these were all I needed, you can figure out 
  % other positioning information)
  \getheightofnode{\pgf@temphei}{pgf@#1-invisible-placement}
  \getwidthofnode{\pgf@tempwid}{pgf@#1-invisible-placement}

  % Copy that into the global vars I use to access them 
  \cs_gset_eq:cN {pgf@#1-height} \pgf@temphei
  \cs_gset_eq:cN {pgf@#1-width} \pgf@tempwid

  % Debug information
  \typeout{__pgf@#1-save-macro_data_:}
  \typeout{_____height_:_\use:c{pgf@#1-height}}
  \typeout{_____width__:_\use:c{pgf@#1-width}}
  \typeout{Ending_placenodelater_for_#1.}
 }

 % Place the node previously specified with the \placenodelater command
 % #1 = identifier for nodes already saved by \placenodelater
 % #2 = location for the node to be placed at
 \DeclareDocumentCommand \placenodenow {d() +m}{
   % Debug
   \typeout{Starting_placenodenow_for_#1.}

   % Save the current transformation
   \pgfgettransform{\pgf@savedtransform}
   % Move the center of the transformation to the placement point specified in
   % #2
   \pgftransformshift{#2}
   % Place all the saved nodes
   \use:c{pgf@#1-data}
   % Reset the transformation
   \pgfsettransform{\pgf@savedtransform}
   % Debug information at time of placement
   \typeout{__#1_data_:}
   \typeout{_____height_:_\use:c{pgf@#1-height}}
   \typeout{_____width__:_\use:c{pgf@#1-width}}
   % Debug
   \typeout{Ending_placenodenow_for_#1.}
 }

 % #1 = Module Name 
 % #2 = Module Label
 % #3 = Submodule definitions
 \DeclareDocumentCommand \nestedmodule {d() +m +m} {

   \ifthenelse{\isundefined\prevnode}
   {
     \tikzstyle{modulepositioning}=[]
   }
   {
     \tikzstyle{modulepositioning}=[
       ,below=of~\prevnode.south~west
       ,anchor=north~west
     ]
   }
   % Create the enclosing node for our module
   \node[
       ,rectangle
       ,modulepositioning
       ,draw,red
   ](#1){
     \tikz[remember~picture]{
       % Create all the submodules before we place them
       \placenodelater(#1-mod-store){ 
         % Create node containing submodules
         \node[
           ,draw,blue
           ,anchor=west
           ,remember~picture
         ](#1modbox){
           \tikz[remember~picture]{
               \let\prevnode\undefined
               % Place all subnodes 
               #3
           }
         };
       }
       \pgfmathsetmacro{\centralheight}{\use:c{pgf@#1-mod-store-height}}
       % Create a main node that's large
         \node[
         ,draw,black
         ,remember~picture
         ,inner~sep=5pt
         ,minimum~width=40pt
         ,minimum~height=\centralheight
       ](#1label){
         #2
       };
 %      % Place sub-modules
       \placenodenow(#1-mod-store){\pgfpointanchor{#1label}{east}}
     }
   };
   % Set self as previous for next node
 %  \cs_set_eq:cN {prevnode} {#1}
     \def\prevnode{#1}
 } 

 \ExplSyntaxOff
 \makeatother
\begin{document}
 \begin{tikzpicture}[remember picture,inner sep =1pt,node distance = 5pt]
     \nestedmodule(root){root}{
       \nestedmodule(port-1){p1}{
         \nestedmodule(port-1-1){p11}{
         };
         \nestedmodule(port-1-2){p12}{
         };
       };
       \nestedmodule(port-2){p2}{
       };
     };
 \end{tikzpicture}
\end{document} 

It is worth noting that there's a number of other issues with this code, including bad usage of latex3 and general lack of robustness. I don't know how well this will work with other PGF/TikZ features, like rotations and transformations.