Here is a solution that does not require introducing new notation. It makes use of align
, plus \left.
and \right.
where necessary, as well as \vphantom
to get the correct sizing of the brackets and parentheses. I also changed the outer parentheses to square brackets because I think it's easier to read, but obviously that's a stylistic choice.
\documentclass{article}
\usepackage{amssymb,amsmath}
\begin{document}
\begin{subequations}\label{eqn:MagneticField3D}
\begin{align}
B_{x}(r,x) = & \frac{N I \mu_{0}}{2\pi}\left[\frac{1}{\sqrt{(a+r)^{2}+(x-d/2)^{2}}}\left(\mathcal{K}(k(r,x-d/2)^{2})
\vphantom{\frac{a^{2}-r^{2}-(x-d/2)^{2}}{(a-r)^{2}+(x-d/2)^{2}}} % to size the ( correctly
\right.\right. \nonumber \\
& \left.\left. +\frac{a^{2}-r^{2}-(x-d/2)^{2}}{(a-r)^{2}+(x-d/2)^{2}}\mathcal{E}(k(r,x-d/2)^{2})\right) \right. \nonumber \\
& \left. -\frac{1}{\sqrt{(a+r)^{2}+(x+d/2)^{2}}}\left(\mathcal{K}(k(r,x+d/2)^{2})
\vphantom{\frac{a^{2}-r^{2}-(x+d/2)^{2}}{(a-r)^{2}+(x+d/2)^{2}}} % to size the ( correctly
\right.\right. \nonumber \\
& \left.\left. +\frac{a^{2}-r^{2}-(x+d/2)^{2}}{(a-r)^{2}+(x+d/2)^{2}}\mathcal{E}(k(r,x-d/2)^{2})\right)
\vphantom{\frac{1}{\sqrt{(a+r)^{2}+(x-d/2)^{2}}}} % to size the ] correctly
\right], \\
B_{r}(r,x) = & \frac{N I \mu_{0}}{2\pi r}\left[\frac{x-d/2}{\sqrt{(a+r)^{2}+(x-d/2)^{2}}}\left(-\mathcal{K}(k(r,x-d/2)^{2})
\vphantom{\frac{a^{2}+r^{2}+(x-d/2)^{2}}{(a-r)^{2}+(x-d/2)^{2}}} % to size the ( correctly
\right.\right. \nonumber \\
& \left.\left. +\frac{a^{2}+r^{2}+(x-d/2)^{2}}{(a-r)^{2}+(x-d/2)^{2}}\mathcal{E}(k(r,x-d/2)^{2})\right) \right. \nonumber \\
& \left. -\frac{x+d/2}{\sqrt{(a+r)^{2}+(x+d/2)^{2}}}\left(-\mathcal{K}(k(r,x+d/2)^{2})
\vphantom{\frac{a^{2}+r^{2}+(x+d/2)^{2}}{(a-r)^{2}+(x+d/2)^{2}}} % to size the ( correctly
\right.\right. \nonumber \\
& \left.\left. +\frac{a^{2}+r^{2}+(x+d/2)^{2}}{(a-r)^{2}+(x+d/2)^{2}}\mathcal{E}(k(r,x-d/2)^{2})\right)
\vphantom{\frac{x-d/2}{\sqrt{(a+r)^{2}+(x-d/2)^{2}}}} % to size the ] correctly
\right].
\end{align}
\end{subequations}
\end{document}

You could also add some \hspace
to bump out the second lines of the "inner" parentheses:
\documentclass{article}
\usepackage{amssymb,amsmath}
\begin{document}
\begin{subequations}\label{eqn:MagneticField3D}
\begin{align}
B_{x}(r,x) = & \frac{N I \mu_{0}}{2\pi}\left[\frac{1}{\sqrt{(a+r)^{2}+(x-d/2)^{2}}}\left(\mathcal{K}(k(r,x-d/2)^{2})
\vphantom{\frac{a^{2}-r^{2}-(x-d/2)^{2}}{(a-r)^{2}+(x-d/2)^{2}}} % to size the ( correctly
\right.\right. \nonumber \\
& \hspace{1cm} \left.\left. +\frac{a^{2}-r^{2}-(x-d/2)^{2}}{(a-r)^{2}+(x-d/2)^{2}}\mathcal{E}(k(r,x-d/2)^{2})\right) \right. \nonumber \\
& \left. -\frac{1}{\sqrt{(a+r)^{2}+(x+d/2)^{2}}}\left(\mathcal{K}(k(r,x+d/2)^{2})
\vphantom{\frac{a^{2}-r^{2}-(x+d/2)^{2}}{(a-r)^{2}+(x+d/2)^{2}}} % to size the ( correctly
\right.\right. \nonumber \\
& \hspace{1cm} \left.\left. +\frac{a^{2}-r^{2}-(x+d/2)^{2}}{(a-r)^{2}+(x+d/2)^{2}}\mathcal{E}(k(r,x-d/2)^{2})\right)
\vphantom{\frac{1}{\sqrt{(a+r)^{2}+(x-d/2)^{2}}}} % to size the ] correctly
\right], \\
B_{r}(r,x) = & \frac{N I \mu_{0}}{2\pi r}\left[\frac{x-d/2}{\sqrt{(a+r)^{2}+(x-d/2)^{2}}}\left(-\mathcal{K}(k(r,x-d/2)^{2})
\vphantom{\frac{a^{2}+r^{2}+(x-d/2)^{2}}{(a-r)^{2}+(x-d/2)^{2}}} % to size the ( correctly
\right.\right. \nonumber \\
& \hspace{1cm} \left.\left. +\frac{a^{2}+r^{2}+(x-d/2)^{2}}{(a-r)^{2}+(x-d/2)^{2}}\mathcal{E}(k(r,x-d/2)^{2})\right) \right. \nonumber \\
& \left. -\frac{x+d/2}{\sqrt{(a+r)^{2}+(x+d/2)^{2}}}\left(-\mathcal{K}(k(r,x+d/2)^{2})
\vphantom{\frac{a^{2}+r^{2}+(x+d/2)^{2}}{(a-r)^{2}+(x+d/2)^{2}}} % to size the ( correctly
\right.\right. \nonumber \\
& \hspace{1cm} \left.\left. +\frac{a^{2}+r^{2}+(x+d/2)^{2}}{(a-r)^{2}+(x+d/2)^{2}}\mathcal{E}(k(r,x-d/2)^{2})\right)
\vphantom{\frac{x-d/2}{\sqrt{(a+r)^{2}+(x-d/2)^{2}}}} % to size the ] correctly
\right].
\end{align}
\end{subequations}
\end{document}

subequations
has nothing to do with the problem – egreg Apr 24 '19 at 14:42\left/right
constructions, manually scale them using the fourbig
constructions, then things can be split. But you should really rewrite this. – daleif Apr 24 '19 at 14:52