You can reduce/eliminate overfull boxes by going \sloppy or with the \sloppypar environment (though \sloppy will still produce underfull boxes). Of course, it eliminates the overfull problem by possibly introducing excessive space.
EDITED to introduce the \opt syntax requested by the OP, though the macro \optpar must still operate on the whole paragraph. Multiple instances of \opt are permitted in the argument to \optpar. The use of an optional argument to \optpar produces a diagnostic result that shows what the two choices would have looked like without sloppy correction.
\optpar employs an auxiliary macro \choosepar{}{} which takes two complete paragraph alternatives as its input. It sets them both \sloppy in a temporary \parbox and compares the total length of text (modulo \textwidth) with the same text set in a simple temporary box (which will do no space stretching).
It picks the paragraph that minimizes the space stretching, and sets it in a sloppypar to eliminate the overfull box, while choosing the one that is minimally bad.
This MWE employs the diagnostic option (shown in red!90) to show the choices at hand.
\documentclass{article}
\usepackage{tabto,ifthen,xcolor}
\def\compressbias{0pt}
\newcommand\savemypos[1]{%
\tabto*{0in}\xdef#1{\TabPrevPos}\tabto{\TabPrevPos}{}%
}
\newcommand\testpar[1]{%
\xdef\myparskip{\the\parindent}%
\setbox2=\hbox{\parbox{\textwidth}{\sloppy%
\hspace*{\myparskip}#1\savemypos{\finalparloc}}}
\setbox4=\hbox{#1}%
\wd4=\dimexpr\wd4-\compressbias\relax%
\def\continuecheck{T}%
\whiledo{\equal{T}{\continuecheck}}{%
\wd4=\dimexpr\wd4-\textwidth\relax%
\ifdim\wd4<\finalparloc\def\continuecheck{F}\fi%
}%
\def\thedelta{\the\dimexpr\finalparloc-\wd4\relax}%
}
\newcommand\choosepar[2]{%
\begin{sloppypar}%
\testpar{#1}\edef\deltaA{\thedelta}%
\testpar{#2}\edef\deltaB{\thedelta}%
\ifdim\deltaA>\deltaB\relax#2\else#1\fi%
\end{sloppypar}%
}
\newcommand\optpar[2][\relax]{\choosepar{%
\def\opt##1##2{##1}#2}{\def\opt##1##2{##2}#2}%
% THE FOLLOWING CODE USES THE OPTIONAL ARGUMENT FOR DIAGNOSTICS
{\color{red!90}\ifx\relax#1\else%
The options had been:\par
\def\opt##1##2{##1}#2\par versus\par
\def\opt##1##2{##2}#2\par
\fi}%
}
\parskip 1ex
\begin{document}
\noindent\hrulefill
\optpar[d]{This \opt{particular}{\ignorespaces} sentence is being
tested, because it has a long box,
\mbox{$\int_0^x A + b\xi + C\xi^2 \,d\xi$}
in the middle of it.}
\noindent\hrulefill
\optpar[d]{This \opt{particular}{small} sentence is
tested with \opt{several}{two} opt calls because it
has a box,
\mbox{$\int_0^x A + b\xi + C\xi^2 \,d\xi$}
in the middle of it.}
\noindent\hrulefill
\optpar[d]{This \opt{particular}{small} sentence is being
tested, because it has a long box,
\mbox{$\int_0^x A + b\xi + C\xi^2 \,d\xi$}
in the middle of it.}
Both choices were bad. But the first option was chosen
because the second choice,
when addressed with sloppy, would introduce more space.
\end{document}
ORIGINAL (pre \optpar) SOLUTION SHOWING SOLUTION ELEMENTS:
In the MWE, I define the two paragraphs, and individually show the badness and their appearance as typeset (I do this to illustrate...it is not required of the method). I then employ \choosepar to select the best one. I then perform in on a second example that contains just one large chuck of boxed material surrounded by normal text.
EDITED to allow the user to select a certain length of what I call \compressbias, so that the modulo math I am using doesn't perceive a small space compression (which is acceptable) as a large space expansion (which has large badness)
\documentclass{article}
\usepackage{tabto,ifthen}
\def\compressbias{0pt}
\newcommand\savemypos[1]{%
\tabto*{0in}\xdef#1{\TabPrevPos}\tabto{\TabPrevPos}{}%
}
\newcommand\testpar[1]{%
\xdef\myparskip{\the\parindent}%
\setbox2=\hbox{\parbox{\textwidth}{\sloppy%
\hspace*{\myparskip}#1\savemypos{\finalparloc}}}
\setbox4=\hbox{#1}%
\wd4=\dimexpr\wd4-\compressbias\relax%
\def\continuecheck{T}%
\whiledo{\equal{T}{\continuecheck}}{%
\wd4=\dimexpr\wd4-\textwidth\relax%
\ifdim\wd4<\finalparloc\def\continuecheck{F}\fi%
}%
\def\thedelta{\the\dimexpr\finalparloc-\wd4\relax}%
}
\newcommand\choosepar[2]{%
\begin{sloppypar}%
\testpar{#1}\edef\deltaA{\thedelta}%
\testpar{#2}\edef\deltaB{\thedelta}%
\ifdim\deltaA>\deltaB\relax#2\else#1\fi%
\end{sloppypar}%
}
\parskip 1ex
\begin{document}
\def\tmpA{%
verylongwordthathasnospace
verylongwordthathasnospace
verylongwordthathasnospace
verylongwordthathasnospace
verylongwordthathasnospace
verylongwordthathasnospace
verylongwordthathasnospace.}%
\testpar{\tmpA}
added space = \thedelta
\begin{sloppypar}
\tmpA
\end{sloppypar}
\def\tmpB{%
very longword that has no space
very longword that has no space
very longword that has no space
very longword that has no space
very longword that has no space
very longword that has no space
very longword that has no space.}%
\testpar{\tmpB}
added space = \thedelta
\begin{sloppypar}
\tmpB
\end{sloppypar}
Now let choosepar pick:
\choosepar{\tmpA}{\tmpB}
EXAMPLE 2:
\choosepar{%
I want to check the quality of this algorithm on a
\mbox{verylongwordthathasnospace}
to see if it will work.}{%
I want to check whether
\mbox{verylongwordthathasnospace}
will fit nicely in this space.}
This was the one rejected:
\begin{sloppypar}
I want to check the quality of this algorithm on a
\mbox{verylongwordthathasnospace}
to see if it will work.
\end{sloppypar}
\end{document}
