A
With markup in the caption text, specifically, \symsf{}
:
Firstly, \ch{A_{$x$}B_{$x-1$} * $n$ H2O}
without \symsf
.

Colour-coding shows the math component is serif; the rest is caption text, in sf style.
Sans version, using \symsf{}
:

but very fiddly in terms of code:
\ch{A_{$\symsf{x}$}B_{$\symsf{x}-\symsf{1}$} * $\symsf{n}$ H2O}
(unicode-math
documentation: 5.1 Math 'style', p 12: "If glyphs are desired that do not map as per the package option [for font style], markup is required")
MWE
\documentclass{book}
\usepackage{xcolor}
\usepackage{amsmath}
\usepackage{unicode-math}
\setmathfont{XITS Math}
\setmainfont{Noto Serif}
\setsansfont{Noto Sans}%[Colour=red]
\setmonofont{Noto Sans Mono}
\setmathfont{Noto Serif}[range=up/{Latin,Greek,num},Colour=blue]
\setmathfont{Noto Serif Italic}[range=it/{latin,greek},Colour=red]
%\setmathfontface\capmath{Asana Math}[Colour=violet]
\usepackage{chemmacros}
\usepackage{caption}
\captionsetup{
labelfont={sf,bf},
textfont={sf},
}
\newcommand{\compound}{\ch{A_{$x$}B_{$x-1$} * $n$ H2O}}
\newcommand{\compoundb}{\ch{A_{$\symsf{x}$}B_{$\symsf{x}-\symsf{1}$} * $\symsf{n}$ H2O}}
\begin{document}
\chapter{}
\rmfamily
Text in rm with math $3a + b_2 = c - \alpha\times x \frac{3}{4}$ and chemical formula \mbox{\compound}.
\sffamily
\noindent Text in sf with math $3a + b_2 = c - \alpha\times x \frac{3}{4}$ and chemical formula \mbox{\compound}.
\ttfamily
\noindent Text in tt with math $3a + b_2 = c - \alpha\times x \frac{3}{4}$ and chemical formula \mbox{\compound}.
\begin{figure}
\begin{center}
xxx
\end{center}
\caption{\compound}
\end{figure}
\begin{figure}
\begin{center}
xxx
\end{center}
\caption{\compoundb}
\end{figure}
\end{document}
B
Alternatively, putting the whole \ch
in math mode instead, and using \mathsf
seems to work:

$\mathsf{\ch{A_{x}B_{x-1} * n H2O}}$
"Seems" is the keyword: $\mathit{\ch{A_{x}B_{x-1} * n H2O}}$
produces the same result.
MWE
\documentclass{book}
\usepackage{xcolor}
\usepackage{amsmath}
\usepackage{unicode-math}
\setmathfont{XITS Math}
\setmainfont{Noto Serif}
\setsansfont{Noto Sans}%[Colour=red]
\setmonofont{Noto Sans Mono}
\setmathfont{Noto Serif}[range=up/{Latin,Greek,num},Colour=blue]
\setmathfont{Noto Serif Italic}[range=it/{latin,greek},Colour=red]
%\setmathfontface\capmath{Asana Math}[Colour=violet]
\usepackage{chemmacros}
\usepackage{caption}
\captionsetup{
labelfont={sf,bf},
textfont={sf},
}
\newcommand{\compound}{\ch{A_{$x$}B_{$x-1$} * $n$ H2O}}
\newcommand{\compoundb}{\ch{A_{$\symsf{x}$}B_{$\symsf{x}-\symsf{1}$} * $\symsf{n}$ H2O}}
\newcommand{\compoundc}{$\mathsf{\ch{A_{x}B_{x-1} * n H2O}}$}
\begin{document}
\chapter{}
\rmfamily
Text in rm with math $3a + b_2 = c - \alpha\times x \frac{3}{4}$ and chemical formula \mbox{\compound}.
\sffamily
\noindent Text in sf with math $3a + b_2 = c - \alpha\times x \frac{3}{4}$ and chemical formula \mbox{\compound}.
\ttfamily
\noindent Text in tt with math $3a + b_2 = c - \alpha\times x \frac{3}{4}$ and chemical formula \mbox{\compound}.
\begin{figure}
\begin{center}
xxx
\end{center}
\caption{\compound}
\end{figure}
\begin{figure}
\begin{center}
xxx with \textbackslash mathsf
\end{center}
\caption{\compoundc}
\end{figure}
\end{document}
C
For this particular \ch
, no-math mode also "works", with results the same as (B).
\ch{A_{x}B_{x-1} * n H2O}
Conclusion
Depending on the ch
, a math mode solution might not be required at all.
The \captionsetup{textfont={sf}}
must be taking place later in the processing, or overriding things, but before the sym..{}
substitutions take place.
This is with \symbf
:
=====
Addendum
The workarounds
The hurdle is that it seems that math and text cannot be truly escaped (yet) inside chemformula
package's \ch
command (chemformula.sty
, line 3568, version 4.15 (2017)).
So the workaround will be manual markup.
There are several ways of doing so.
Given that two fonts have to intersect

math sans upright (sfup) - marked in red in the image - and text sffamily - marked in blue -
\documentclass{article}
\usepackage{xcolor}
\usepackage{amsmath}
\usepackage{unicode-math}
\setmathfont{XITS Math}
\setmainfont{Noto Serif}
\setsansfont{Noto Sans}[Colour=blue]
\setmonofont{Noto Sans Mono}
\setmathfont{Noto Sans}[range=sfup/{latin,greek,num},Colour=red]
\usepackage{chemformula}%3.3 own manual; separate from chemmacros since Jul-2013
\usepackage{caption}
\captionsetup{
labelfont={sf,bf},
textfont={sf},
}
\newcommand{\compound}{\ch{A_{$x$}B_{$x-1$} * $n$ H2O}}
\newcommand{\compoundsfup}{\ch{A_{$\symsfup{x}$}B_{$\symsfup{x}-\symsfup{1}$} * $\symsfup{n}$ H2O}}
\begin{document}
Inline text with \compound .
\vspace{24pt}
\begin{tabular}{l|ll}
Text & $\symsfup{symsfup}$ & normal \\
\hline \\
\rmfamily rmfamily & \rmfamily\compoundsfup & \rmfamily\compound \\
\sffamily sffamily & \sffamily\compoundsfup & \sffamily\compound\\
\ttfamily ttfamily & \ttfamily\compoundsfup & \ttfamily\compound \\
\end{tabular}
\end{document}
markup can be done totally manually, but is time-consuming, and two separate markups have to be maintained:
Method 1

\newcommand{\compound}{\ch{A_{$x$}B_{$x-1$} * $n$ H2O}}
and
\newcommand{\compoundsfup}{\ch{A_{$\symsfup{x}$}B_{$\symsfup{x}-\symsfup{1}$} * $\symsfup{n}$ H2O}}
Code
\documentclass{article}
\usepackage{regexpatch}
\usepackage{xcolor}
\usepackage{amsmath}
\usepackage{unicode-math}
\setmathfont{XITS Math}
\setmainfont{Noto Serif}
\setsansfont{Noto Sans}[Colour=blue]
\setmonofont{Noto Sans Mono}
\setmathfont{Noto Sans}[range=sfup/{latin,greek,num},Colour=red]
\usepackage{chemformula}%3.3 own manual Jul-2013
\usepackage{caption}
\captionsetup{
labelfont={sf,bf},
textfont={sf},
}
\newcommand{\compound}{\ch{A_{$x$}B_{$x-1$} * $n$ H2O}}
\newcommand{\compoundsfup}{\ch{A_{$\symsfup{x}$}B_{$\symsfup{x}-\symsfup{1}$} * $\symsfup{n}$ H2O}}
\begin{document}
Inline text with \compound .
\begin{figure}
\begin{center}
xxx
\end{center}
\caption{\compoundsfup}
\end{figure}
Two separate markups.
\end{document}
Method 2
Slightly more efficient is parsing twice from the source code text, using regex to automatically replace anything between $...$
with symbfup
in one case and symrm
in the other:

The regex is: \$([^\$]+)\$
= one or more of anything after a $
, that is not a $
, up to the next $
, treated as a group.
\documentclass{article}
\usepackage{xcolor}
\usepackage{amsmath}
\usepackage{unicode-math}
\setmathfont{XITS Math}
\setmainfont{Noto Serif}
\setsansfont{Noto Sans}[Colour=blue]
\setmonofont{Noto Sans Mono}
\setmathfont{Noto Sans}[range=sfup/{latin,greek,num},Colour=red]
\usepackage{chemformula}%3.3 own manual Jul-2013
\usepackage{caption}
\captionsetup{
labelfont={sf,bf},
textfont={sf},
}
\ExplSyntaxOn
\NewDocumentCommand{\cmathrm}{m}
{
\tl_set:Nn \l_lyssane_cmathrm_tl { #1 }
% change every run of anything between $ ...$ into $sym...{}$
\regex_replace_all:nnN
{ \$([^\$]+)\$ }
{ \c{ensuremath}\cB\{\c{symrm}\cB\{\1\cE\}\cE\} }
\l_lyssane_cmathrm_tl
% print the result
\tl_use:N \l_lyssane_cmathrm_tl
}
\NewDocumentCommand{\cmathsf}{m}
{
\tl_set:Nn \l_lyssane_cmathsf_tl { #1 }
% change every run of anything between $ ...$ into $sym...{}$
\regex_replace_all:nnN
{ \$([^\$]+)\$ }
{ \c{ensuremath}\cB\{\c{symsfup}\cB\{\1\cE\}\cE\} }
\l_lyssane_cmathsf_tl
% print the result
\tl_use:N \l_lyssane_cmathsf_tl
}
\ExplSyntaxOff
\newcommand\compound{\ch{\cmathrm{A_{$2x$}B_{$x$-$1$} * $n$ H2O}}: roman h1.}
\newcommand\compoundcap{\ch{\cmathsf{A_{$2x$}B_{$x$-$1$} * $n$ H2O}}: sf h2.}
\begin{document}
{\rmfamily\compound} ;
{\sffamily\compoundcap}
\begin{figure}
\begin{center}
xxx
\end{center}
\caption{\compoundcap}
\end{figure}
\end{document}
(acknowledgement: the l3 code is adapted from egreg's solution for transliterating Hittite)
\ch
loses a space here, somehow, after the n
of $n$
. Easy enough to re-insert one. (Edited to add: the reason is that the parser regex step replaces $n$ H
with \ensuremath{symsf{n}}H
, so \ch
never gets the opportunity to add the math space. Oops!)
This method requires the input code to be given twice, once to each parser.
\newcommand\compound{\ch{\cmathrm{A_{$2x$}B_{$x$-$1$} * $n$ H2O}}: roman h1.}
and
\newcommand\compoundcap{\ch{\cmathsf{A_{$2x$}B_{$x$-$1$} * $n$ H2O}}: sf h2.}
Method 3
And widening-up the $..$
pairs to include the __
so that the parsed text can be typeset independently of \ch
,
\cmathrm{A$_{2x}$B$_{x-1}$ * $n$ H2O} i2a;
\cmathsf{A$_{2x}$B$_{x-1}$ * $n$ H2O} i2b.
shows that \ch
is seeing sf text (blue), rather than sf math (red).

Which is OK, because it is all sf now anyway, and, in turn, the sf in the caption comes out OK.
\documentclass{article}
\usepackage{regexpatch}
\usepackage{xcolor}
\usepackage{amsmath}
\usepackage{unicode-math}
\setmathfont{XITS Math}
\setmainfont{Noto Serif}
\setsansfont{Noto Sans}[Colour=blue]
\setmonofont{Noto Sans Mono}
\setmathfont{Noto Sans}[range=sfup/{latin,greek,num},Colour=red]
\usepackage{chemformula}%3.3 own manual Jul-2013
\usepackage{caption}
\captionsetup{
labelfont={sf,bf},
textfont={sf},
}
\ExplSyntaxOn
\NewDocumentCommand{\cmathrm}{m}
{
\tl_set:Nn \l_lyssane_cmathrm_tl { #1 }
% change every run of anything between $ ...$ into $sym...{}$
\regex_replace_all:nnN
{ \$([^\$]+)\$ }
{ \c{ensuremath}\cB\{\c{symrm}\cB\{\1\cE\}\cE\} }
\l_lyssane_cmathrm_tl
% print the result
\tl_use:N \l_lyssane_cmathrm_tl
}
\NewDocumentCommand{\cmathsf}{m}
{
\tl_set:Nn \l_lyssane_cmathsf_tl { #1 }
% change every run of anything between $ ...$ into $sym...{}$
\regex_replace_all:nnN
{ \$([^\$]+)\$ }
{ \c{ensuremath}\cB\{\c{symsfup}\cB\{\1\cE\}\cE\} }
\l_lyssane_cmathsf_tl
% print the result
\tl_use:N \l_lyssane_cmathsf_tl
}
\ExplSyntaxOff
\begin{document}
\begin{figure}
\begin{center}
xxx i2
\end{center}
\caption{\ch{\cmathsf{A$_{2x}$B$_{x-1}$ * $n$\ H2O}} i2b.}
\end{figure}
Normal text, without \textbackslash ch:
\cmathrm{A$_{2x}$B$_{x-1}$ * $n$ H2O} i2a;
\cmathsf{A$_{2x}$B$_{x-1}$ * $n$ H2O} i2b.
\vspace{24pt}
rmfamily and sffamily, with \textbackslash ch:
{\rmfamily\ch{\cmathrm{A$_{2x}$B$_{x-1}$ * $n$ H2O}} i2a rmf};
{\sffamily\ch{\cmathsf{A$_{2x}$B$_{x-1}$ * $n$ H2O}} i2b sff}. {\small <= needs a space}
\end{document}
Method 4 (non-working)
Putting the input code into one macro requires one level of appropriate suppression of expansion (or escaped code) to prevent the parsers and \ch
getting an already tokenized list.

\documentclass{article}
\usepackage{xcolor}
\usepackage{amsmath}
\usepackage{unicode-math}
\setmathfont{XITS Math}
\setmainfont{Noto Serif}
\setsansfont{Noto Sans}[Colour=blue]
\setmonofont{Noto Sans Mono}
\setmathfont{Noto Sans}[range=sfup/{latin,greek,num},Colour=red]
%\usepackage{chemmacros}
\usepackage{chemformula}%3.3 own manual Jul-2013
\usepackage{caption}
\captionsetup{
labelfont={sf,bf},
textfont={sf},
}
\newcommand\compoundi{A$_{2x}$B$_{x-1}$ * $n$ H2O}
%\edef\compoundia{\meaning\compoundi}
\ExplSyntaxOn
\NewDocumentCommand{\cmathrm}{m}
{
\tl_set:Nn \l_lyssane_cmathrm_tl { #1 }
% change every run of anything between $ ...$ into $sym...{}$
\regex_replace_all:nnN
{ \$([^\$]+)\$ }
{ \c{ensuremath}\cB\{\c{symrm}\cB\{\1\cE\}\cE\} }
\l_lyssane_cmathrm_tl
% print the result
\tl_use:N \l_lyssane_cmathrm_tl
}
\NewDocumentCommand{\cmathsf}{m}
{
\tl_set:Nn \l_lyssane_cmathsf_tl { #1 }
% change every run of anything between $ ...$ into $sym...{}$
\regex_replace_all:nnN
{ \$([^\$]+)\$ }
{ \c{ensuremath}\cB\{\c{symsfup}\cB\{\1\cE\}\cE\} }
\l_lyssane_cmathsf_tl
% print the result
\tl_use:N \l_lyssane_cmathsf_tl
}
\ExplSyntaxOff
\begin{document}
%\begin{figure}
%\begin{center}
%xxx f
%\end{center}
%\caption{\ch{\cmathb{$\mathsf{A}_{|2x|}\mathsf{B}_{|x|-|1|}$ * |n| H2O}} f}
%\end{figure}
\compoundi cmpd i -- the macro ;
{\rmfamily\ch{\compoundi} cmpd i rm -- with \textbackslash ch };
{\sffamily\ch{\compoundi} cmpd i sf -- with \textbackslash ch }.
{\rmfamily\ch{\cmathrm{\compoundi}} cmpd i rm cm -- and parser};
{\sffamily\ch{\cmathsf{\compoundi}} cmpd i sf cm -- and parser}.
\end{document}