# Underbrace under square root sign plain TeX $$\rlap{\sqrt{\phantom{\displaystyle\pmatrix{A^{-1} b(x)\cr -1}^T \pmatrix{A & b(x) \cr b(x)^T & \Phi(x,x)} \pmatrix{A^{-1} b(x) \cr -1}}}} \mskip14mu \displaystyle\pmatrix{A^{-1} b(x)\cr -1}^T \underbrace{\displaystyle \pmatrix{A & b(x) \cr b(x)^T & \Phi(x,x)}}% _A_{\Phi, X\cup\{x\}}} \pmatrix{A^{-1} b(x) \cr -1}$$

\bye


Is there a better way to obtain the same effect, without having to repeat the formula and manually adding a space (\mskip14mu)?

You could (a) apply a \smash instruction to the \underbrace{...}_{...} part and (b) insert a \vphantom directive after the \sqrt{...} part. $$\sqrt{ \pmatrix{A^{-1} \cr -1}^{\!T} \smash{\underbrace{\pmatrix{A & b(x) \cr b(x)^T & \Phi(x,x)}}% _{\textstyle A_{\Phi, X\cup\{x\}}}} \pmatrix{A^{-1} b(x) \cr -1} } \vphantom{\underbrace{\pmatrix{.\cr .}}_{\textstyle A_{\Phi}}}$$
\bye


Observe that I've also dropped some \displaystyle directives and have converted others to \textstyle.

TeX can measure the object:

$$\def\contentsA{\pmatrix{A^{-1} b(x) \cr -1}} \def\contentsB{\pmatrix{A & b(x) \cr b(x)^T & \Phi(x,x)}} \setbox0=\hbox{\displaystyle \contentsA^{\!T} {\underbrace{\!\contentsB\!}_A_{\Phi, X\cup\{x\}}}} \contentsA } \sqrt{\vphantom{\contentsA^T}\hphantom{\copy0}} \kern-\wd0 \box0$$

\bye I made a couple of changes: braces like {\underbrace{...}_{...}} in order to avoid it being considered a math operator with an excess of spacing; for the same reason I introduced a couple of \! tokens. And also one for the exponent in order to place it nearer the top of the parenthesis.