Minimal working example

\documentclass[11pt, twoside]{report}
\usepackage{caption}
\usepackage[]{subfig}
\usepackage{graphicx}

\begin{document}
\begin{figure}
\captionsetup[subfigure]{format=hang,singlelinecheck=false,justification=raggedright}
\subfloat[4-Hy\-droxy\-ben\-zo\-ic acid]{
\includegraphics[width=0.2\textwidth]{4hba_600dpi.png}
}
\hfill
\subfloat[Hy\-droxy\-ben\-zo\-ic acid]{
\includegraphics[width=0.2\textwidth]{4hba_600dpi.png}
}
\hfill
\subfloat[Hydroxybenzoic acid]{
\includegraphics[width=0.2\textwidth]{4hba_600dpi.png}
}

\subfloat[Hydroxybenzoic acid]{
\includegraphics[width=0.2\textwidth]{4hba_600dpi.png}
}
\end{figure}

\begin{figure}
\captionsetup[subfigure]{format=hang,singlelinecheck=false,justification=}
\subfloat[4-Hy\-droxy\-ben\-zo\-ic acid]{
\includegraphics[width=0.2\textwidth]{4hba_600dpi.png}
}
\hfill
\subfloat[Hy\-droxy\-ben\-zo\-ic acid]{
\includegraphics[width=0.2\textwidth]{4hba_600dpi.png}
}
\hfill
\subfloat[Hydroxybenzoic acid]{
\includegraphics[width=0.2\textwidth]{4hba_600dpi.png}
}

\subfloat[Hydroxybenzoic acid]{
\includegraphics[width=0.2\textwidth]{4hba_600dpi.png}
}
\end{figure}
\end{document}

Description

When the justification 'raggedright' is used and the caption must be hyphenated another break is introduced at the next space. Non-breaking spaces can prevent this, but are a poor solution. The workaround used here is resetting justification to the default value. At least, I assume it's what I'm doing in the second figure. The document was compiled using xelatex.

Question(s)

Am I doing anything wrong? Is this a bug? Am I missing a setting? How can I have raggedright justification and proper line-breaking of hyphenated words in subfig captions?

Use justification=RaggedRight (it might require two LaTeX runs). Be careful with end-of-lines.

\documentclass[11pt, twoside]{report}

\usepackage{caption}
\usepackage{subfig}
\usepackage{ragged2e}
\usepackage[demo]{graphicx}

\begin{document}

\begin{figure}
\captionsetup[subfigure]{format=hang,singlelinecheck=false,justification=RaggedRight}

\subfloat[4-Hy\-droxy\-ben\-zo\-ic acid]{%
\includegraphics[width=0.2\textwidth]{4hba_600dpi.png}%
}\hfill
\subfloat[Hy\-droxy\-ben\-zo\-ic acid]{%
\includegraphics[width=0.2\textwidth]{4hba_600dpi.png}%
}\hfill
\subfloat[Hydroxybenzoic acid]{%
\includegraphics[width=0.2\textwidth]{4hba_600dpi.png}%
}

\subfloat[Hydroxybenzoic acid]{%
\includegraphics[width=0.2\textwidth]{4hba_600dpi.png}%
}

\end{figure}

\end{document}