Here is a possible way to accomplish what you want. It is a little involved, but it seems to generally work well as long as the wrapped expression has at least a certain height/depth. It may however require some more testing and can probably be made more efficient.
I'm defining \unevenbrace
and \unevenbrack
below.
Both take one argument and wrap it in unbalanced braces or brackets respectively, as requested.
Note that everything scales properly when these macros are used in sub-/superscripts.
\documentclass{article}
\usepackage{amsmath}
\usepackage{trimclip} %% <- for \clipbox
\makeatletter %% <- make @ usable in command names
\newcommand*\unevendelim[3]{{\mathpalette\unevendelim@{{#1}{#2}{#3}}}}
\def\unevendelim@#1#2{\unevendelim@@{#1}#2}%
\def\unevendelim@@#1#2#3#4{%
\sbox0{$\m@th#1#4$}%
\sbox6{$\m@th#1\{\}$}%
\unevendelim@@@{#1}{\left#2}{\right.}%
\copy0
\unevendelim@@@{#1}{\left.}{\right#3}%
}
\def\unevendelim@@@#1#2#3{%
\sbox2{$\m@th#1#2\rule{0pt}{\ht0}#3$}%
\sbox4{$\m@th#1#2\rule[-\dp0]{0pt}{\dp0}#3$}%
\ifdim\ht2>\ht4
\ooalign{\clipbox{0pt {\dimexpr\dp2-\dp6\relax} 0pt 0pt}{\copy2}\cr
\raisebox{-\dp6}{\clipbox{0pt 0pt 0pt {\dimexpr2\ht2-\ht4+\dp6}}{\copy2}}\cr}
\else
\ooalign{\raisebox{\ht6}{\clipbox{0pt {\dimexpr2\dp4-\dp2+\ht6} 0pt 0pt}{\copy4}}\cr
\clipbox{0pt 0pt 0pt {\dimexpr\ht4-\ht6}}{\copy4}\cr}
\fi
}
\makeatother %% <- revert @
% \newcommand*\unevenparen{\unevendelim()} %% <- looks AWFUL
\newcommand*\unevenbrace{\unevendelim\{\}}
\newcommand*\unevenbrack{\unevendelim[]}
\begin{document}
\[
U_{ij}^{\mathrm{tor}}(\hat{\mathbf{r}}_{ij},\mathbf{\Omega}_i,\mathbf{\Omega}_j)
= \exp\unevenbrace{
- \frac{ \unevenbrack{
\min\limits_{\phi^{\mathrm{offset}}_{\alpha\beta}}
(\phi_{\alpha\beta}-\phi^{\mathrm{offset}}_{\alpha\beta} )
}^{2}
}{2\sigma_{\mathrm{tor}}^{2}}
}
\]
\[
U_{ij}^{\mathrm{tor}}(\hat{\mathbf{r}}_{ij},\mathbf{\Omega}_i,\mathbf{\Omega}_j)
= \exp\unevenbrace{
- \frac{2\sigma_{\mathrm{tor}}^{2}
}{ \unevenbrack{
\min\limits^{\phi^{\mathrm{offset}}_{\alpha\beta}}
(\phi_{\alpha\beta}-\phi^{\mathrm{offset}}_{\alpha\beta} )
}^{2}
}
}
\]
\[
e^{e^{\exp\unevenbrace{
- \frac{ \unevenbrack{
\min\limits_{\phi^{\mathrm{offset}}_{\alpha\beta}}
(\phi_{\alpha\beta}-\phi^{\mathrm{offset}}_{\alpha\beta} )
}^{2}
}{2\sigma_{\mathrm{tor}}^{2}}
}
}}
\]
\end{document}

These commands work by stitching together different parts of the braces/brackets produced with \left
and \right
using \clipbox
. It doesn't work for parentheses because these don't have a straight part where this suture can be applied unnoticed.
Really quick explanation
Here's what happens, step by step:
\mathpalette
is used to get the math style (display/text/script/scriptscript) right.
<expr>
is typeset in the right style in box register 0
;
\{\}
is similarly typeset in box register 6
, just so we can determine its height/depth (which feels a little gratuitous).
\left\{<something as tall as <expr>>\right.
is typeset in box 2
.
\left.<something as deep as <expr>>\right\}
is typeset in box 4
.
- If box
2
is taller, then its top half should be drawn, along with however much is needed from the bottom to reach the depth of box 4
. The "top half" is defined as everything above the depth of box 6
.
- If box
4
is taller, then its bottom half should be drawn, along with however much is needed from the top to reach the height of box 2
. The "bottom half" is defined as everything below the height of box 6
.
\clipbox
is used to cut up box 2
and box 4
. This command is defined by the trimclip
package, which is part of adjustbox
.
\raisebox
is used to get the vertical positioning right.
\ooalign
is used to splice these cuttings together.
- The contents of box
0
(which was <expr>
) is printed.
- Steps 4–10 are repeated for the right brace.
{}
as your fences, there is no reason to scale them this big, even in smaller sizes it is clear what the{}
fence in. (3) I would how ever rewrite this and pull the fraction out front, then the problem disappears.