How to auto linebreak before/after parenthesis in inline math

Question

I read this answer "Allowing line break at ',' in inline math mode" breaks citations

I want to change ('s catcode automatically to avoid breaking some packages.

\documentclass[a5paper]{ltjsarticle}

\usepackage{etoolbox}
\usepackage{amsmath, amssymb}
\usepackage[math-style=TeX]{unicode-math}

\makeatletter
\def\old@lparen{(}
\catcode`\(=13
\def({%
  \ifmmode%
    \discretionary{}{}{}\old@lparen%
  \else% for \mbox, \text, etc...
    \old@lparen%
  \fi}
\catcode`\(=12
\makeatother
\preto{\(}{\catcode`\(=13}{}{}
\appto{\)}{\catcode`\(=12}{}{}

\begin{document}
\renewcommand{\mid}{\mathopen{}\,|\,\mathclose{}}

\hrulefill

\((A \mid B),(A \mid B),(A \mid B),(A \mid B),(A \mid B),(A \mid B),(A \mid B),(A \mid B),(A \mid B),(A \mid B),(A \mid B)\)

$(A \mid B),(A \mid B),(A \mid B),(A \mid B),(A \mid B),(A \mid B),(A \mid B),(A \mid B),(A \mid B),(A \mid B),(A \mid B)$

\end{document}

\( and \) are easily patched with etoolbox but $ and $ are stubbron for me...

Update

I want a break after , (highest priority). And I want a break before ( or after )

Update2

I thought "a break before ( or after )" is equivalent to "nolinebreak before/after \mid and operands (in this case)". But this idea is not correct?

"Allowing line break at ',' in inline math mode" breaks citations this page's trick makes linebreak in inner parenthesis (of course, this is normal behavior).

egreg's suggestion looks great except linebreaking rparen followed comma.

I want mixed soulution.

Answer 1

I'm not sure you want a break before (, but rather a possible break after ).

\documentclass[a5paper]{ltjsarticle}

\usepackage{etoolbox}
\usepackage{amsmath, amssymb}
\usepackage[math-style=TeX]{unicode-math}

\begingroup\lccode`~=`)
\AtBeginDocument{%
  \edef\rparen{\Umathcharnum\the\Umathcodenum`) }
  \mathcode`)=\string"8000
}
\lowercase{\endgroup\def~}{\rparen\penalty0 }

\begin{document}

\noindent\hrulefill

\(
 (A \mid B)(A \mid B)(A \mid B)(A \mid B)(A \mid B)(A \mid B)
 (A \mid B)(A \mid B)(A \mid B)(A \mid B)(A \mid B)
\)

$(A \mid B)(A \mid B)(A \mid B)(A \mid B)(A \mid B)(A \mid B)
 (A \mid B)(A \mid B)(A \mid B)(A \mid B)(A \mid B)$

\end{document}

Note that this also allows using \right), while your approach would make \left( illegal.

The trick is making ) math active and not active.