lmodern package reduces the size of automatically sized delimiters in math mode

Including lmodern package reduces the size of \left and \right delimiters. For example, try this with and without lmodern:

%!TEX encoding = UTF-8 Unicode

\documentclass[a4paper,12pt]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}

\usepackage{lmodern}  % <<<

\usepackage{amsmath}

\begin{document}
\begin{gather*}
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)\\
\left(\frac{1}{x}\right)
\end{gather*}
\end{document}


Is this a feature or a bug? What it the "correct" behaviour? If this is a bug, is there a fix?

Note that Latin Modern used in XeLaTeX does not reduce the size of delimiters, but using lmodern package with XeLaTeX also reduces the size of delimiters.

One fraction is sufficient to show the problem.

Load the fixcmex package and consult its documentation to see where the problem lies.

\documentclass[a4paper,12pt]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{amsmath}

\usepackage{lmodern}  % <<<
\usepackage{fixcmex}

\begin{document}

$\left(\frac{1}{x}\right)\sum$

\end{document}


For comparison, here's the output without fixcmex; as you see, not only the parentheses are wrong, also the summation sign is smaller.