It's a case of different hats, or parallel threads.
The idea of families might be more accessible if thought of as sets of (already-formatted) glyphs within a font.
In text, x
(the glyph U+0078 LATIN SMALL LETTER X) can be typeset as upright, or italic or bold etc, according to the font file(s).
In math, 𝑥 and 𝒙 are two additional, different, glyphs in the same font, and they belong to different sets ("alphabet families"), mathematical italic, and mathematical bold italic: U+1D465 MATHEMATICAL ITALIC SMALL X = (𝑥) and U+1D499 MATHEMATICAL BOLD ITALIC SMALL X = (𝒙)
Which means that it should be possible to map elements from one set to another (and also across fonts, too).
unicode-math
package allows mapping using the ->
operator of the range=
option.
Here, the same font is mapped to itself, specifically, math bold italic is replaced by math italic which in turn has been doubled in size and made blue, for clarity. Math bold italic being accessed via the \symbfit{}
command.

MWE
\documentclass[12pt]{article}
\usepackage{xcolor}
\usepackage{amsmath,amssymb}
\usepackage{unicode-math}
\setmainfont{Liberation Serif}
\newfontface\fmlmm{Latin Modern Math}
\setmathfont{Latin Modern Math}
\setmathfont{Latin Modern Math}[range=bfit->it,Scale=2,Colour=blue]
%-----------
\begin{document}
U+1D465 MATHEMATICAL ITALIC SMALL X = ({\fmlmm 𝑥})
U+1D499 MATHEMATICAL BOLD ITALIC SMALL X = ({\fmlmm 𝒙})
$ x + x+y + y$
$ x + \symbfit{x}+\symbfit{y} + y$
\end{document}
Alternatively, a lower-level method, but still using a scaled-version of the font as symbol-source, can be by assigning a \Umathcode
to a glyph (red, below, accessed via direct input, or unicode-math symbol name, or unicode codepoint value), or obtained by defining a formatted custom command (green).

MWE
\documentclass[12pt]{article}
\usepackage{xcolor}
\usepackage{amsmath,amssymb}
\usepackage{unicode-math}
\setmainfont{Liberation Serif}
\setmathfont{XITS Math}
\newfontfamily\fmymath{XITS Math}[Colour=red,Scale=2,NFSSFamily=mymath]
\newfontfamily\fmymathg{XITS Math}[Colour=green,Scale=2]
\DeclareSymbolFont{bigletters}{\encodingdefault}{mymath}{m}{it}
\newcommand{\makebigmathletter}[1]{%
\begingroup\lccode`a=#1\lowercase{\endgroup
\Umathcode`a}="0 \csname symbigletters\endcsname\space #1
}
%\count255="409
%\loop\ifnum\count255<"44F
% \advance\count255 by 1
% \makebigmathletter{\count255}
%\repeat
\makebigmathletter{"1D465}
\makebigmathletter{"1D466}
%-----------
\newcommand{\declaresymbols}[2]{\expandafter\newcommand\csname#1big\endcsname{%
\text{\fmymathg#2}}}
\declaresymbols{x}{\symbol{"1D465}}
\declaresymbols{y}{\symbol{"1D466}}
%-----------
\begin{document}
\[
x + 𝑥 +\mitx + \symbol{"1D466} + y +\xbig + \ybig
\]
\end{document}
What the start of Unicode's Mathematical Alphanumeric Symbols codeblock looks like in BabelMap:

996 characters!
So, to take a step towards trying to answer the question, the size/shape/weight levers for the single-thread in text mode has resulted in a different approach to the many-levered multi-threads of math mode. They are almost orthogonal in approach.
Italics (and bold and Fraktur etc) in math mode has semantic meaning; text mode, it's more a style or decoration; if text mode had retained its original meaning for italics (names of people, places and things), for example, then perhaps likely every font would have italic x and upright x as separate glyphs.