Is it that the float package begins to show its age? I cannot get it to work right with XeLaTeX. Here is an example:
\documentclass{article} % compile with xelatex
\usepackage{float}
\usepackage{polyglossia}
\setmainlanguage{hebrew}
\setotherlanguage{english}
\newfontfamily\hebrewfont[Script=Hebrew]{Ezra SIL}% or{SBL Hebrew}
\newfloat{thingy}{htbp}{lot}[section]
\begin{document}
\begin{thingy}
\[ E = m c^2 \]
\begin{hebrew}\setRTL\caption{This caption should be set RTL}\end{hebrew}
\end{thingy}
\end{document}

floatis processing the\caption's in general. Apparently it stores the caption and places it depending on the layout of this float time. This also makes the position of\captionirrelevant. – Martin Scharrer♦ Feb 14 '11 at 7:46