This is a good example why I usually discourage using \left
and \right
indiscriminately.
With \DeclarePairedDelimiter\norm{\lVert}{\rVert}
you are defining the macro \norm
that can be used in the following ways:
\norm{x}
\norm[\big]{x}
\norm[\Big]{x}
\norm[\bigg]{x}
\norm[\Bigg]{x}
\norm*{x}
The simple call does nothing to the size of the delimiters, the following ones with the optional argument specify the size “manually”, the last one tells TeX to use “automatic” sizing.
The trick you're using tells TeX that if you call \norm{x}
, \oldnorm*{x}
is executed and, when \norm*{x}
is found, \oldnorm{x}
is actually executed.
Here's how you could do it:
\documentclass{article}
\usepackage{mathtools}
\DeclarePairedDelimiter\norm{\lVert}{\rVert}%
\makeatletter
\let\oldnorm\norm
\def\norm{\@ifstar{\oldnorm}{\oldnorm*}}
\makeatother
\begin{document}
(1) $\norm{a} - \norm{b_{t|t}}$
\bigskip
(2) $\norm*[\big]{a} - \norm*[\big]{b_{t|t}}$
\bigskip
(3) $\norm*{a} - \norm*{b_{t|t}}$
\end{document}

I have no doubt whatsoever that (3) is the only correct way. The trick you're using with \oldnorm
is simply useless: just remove it and type
\documentclass{article}
\usepackage{mathtools}
\DeclarePairedDelimiter\norm{\lVert}{\rVert}%
\begin{document}
(3) $\norm{a} - \norm{b_{t|t}}$
\end{document}
using the optional argument or the *-form only when they are really necessary.
This input will give the form (3) above.
\norm[\big]{…}
, for instance.\norm
then you save that definition as\oldnorm
then you define\norm
to beoldnorm
so get back to where you were.\norm
and\norm*
: it's\norm*[\big]{…}
you have to write.