Scaling in general does not affect fontsize, but just the size of the elements drawn by tikz. Fontsize is always defined by your documentclass, if you don't explicitly override their setting. Consider the following example:
\documentclass[tikz, border=2mm]{standalone}
%
\begin{document}
%
\begin{tikzpicture}
\draw (0,0) rectangle (2,1) node [midway] {Test};
\draw [gray, dashed] (0.5,0.25) rectangle (1.5,0.75) (-1,-0.5) rectangle (3,1.5);
\end{tikzpicture}
%
\begin{tikzpicture}[scale=0.5]
\draw (0,0) rectangle (2,1) node [midway] {Test};
\draw[white] (-3,-1.5) rectangle (5,2.5);% just for scaling the standalone page correctly
\end{tikzpicture}
%
\begin{tikzpicture}[scale=2]
\draw (0,0) rectangle (2,1) node [midway] {Test};
\end{tikzpicture}
\end{document}
Here I draw in the first tikzpicture three rectangles and insert text in the middle. The inner and the outer rectangles marked gray and dashed would correspond to a scaling of 0.5 and 2 respectively of the black solid one. I then take the definition of the solid rectangle and draw them in the next two tikzpictures with scaling 0.5 and 2 and as you can see, just the size of the ractangle changes, but the fontsize stays the same.

So the consequence is, that scaling the fontsize dependent on the tikz elements doesn't really work, but you can define the elements with respect to the fontsize, i.e. in units of em (1 em is actually the font size)
Then you could for example say:
\documentclass[tikz, border=2mm]{standalone}
%
\begin{document}
%
\begin{tikzpicture}
\draw (0em,0em) rectangle (4em,2em) node [midway] {Test};
\draw [gray, dashed] (1em,0.5em) rectangle (3em,1.5em) (-2em,-1em) rectangle (6em,3em);
\end{tikzpicture}
%
\begin{tikzpicture}[scale=0.5]
\draw (0em,0em) rectangle (4em,2em) node [midway] {Test};
\draw[white] (-6em,-3em) rectangle (10em,5em);% just for scaling the standalone page correctly
\end{tikzpicture}
%
\begin{tikzpicture}[scale=2]
\draw (0em,0em) rectangle (4em,2em) node [midway] {Test};
\end{tikzpicture}
\end{document}
So in the first picture we again define a rectangle, now of double the fontsize in height and four times the fontsize in width. The dashed, gray ones agian would correspond to scaling 2 and 0.5. When we now scale the initial rectangle by 0.5 (second picture), we know then, that it ends up with fontsize in height and double the fontsize in width, and as you see it snugly fits. Whereas a scaling of 2 gives us four times fontsize height and eight times in width. So if you want to fit your font into your boxes, you actually have to fit the boxes to the font!

And last but not least the evidence, that a change in fontsize also changes the the drawings accordingly, which where defined in units of em. I draw an outer rectangle of size 2 cm by 1 cm in comparison:
\documentclass[tikz, border=2mm]{standalone}
%
\begin{document}
%
\begin{tikzpicture}
\draw (0em,0em) rectangle (2em,1em) node [midway] (a) {Test};
\node[draw, shape=rectangle, minimum width= 2cm, minimum height=1cm, anchor=center] at (a) {};
\end{tikzpicture}
%
{\huge
\begin{tikzpicture}
\draw (0em,0em) rectangle (2em,1em) node [midway] (a) {Test};
\node[draw, shape=rectangle, minimum width= 2cm, minimum height=1cm, anchor=center] at (a) {};
\end{tikzpicture}
}
%
{\tiny
\begin{tikzpicture}
\draw (0em,0em) rectangle (2em,1em) node [midway] (a) {Test};
\node[draw, shape=rectangle, minimum width= 2cm, minimum height=1cm, anchor=center] at (a) {};
\end{tikzpicture}
}
\end{document}
