For mathematical documents i like using the fourier package with the upright option, however the symbols \mathcal produces are quite too quirky in my opinion, so I replaced them with the default font via \DeclareMathAlphabet{\mathcal}{OMS}{cmsy}{m}{n} - however these symbols have a different font size than the fourier fonts.
Here is an example which shows that the caligraphic X is too big:

How can I reduce the size of \mathcal symbols?
