After some experiments and tests, I managed to create the following code for \upbracefill and \downbracefill in Plain XeTeX, which it seems is that used for \underbrace and \overbrace:
\def\midshift#1{
\setbox0=\hbox{#1}\dimen0=\ht0\advance\dimen0by+\dp0\advance\dimen0by-1ex
\lower.5\dimen0\box0 }
\def\rotatebrace#1{%
\leavevmode\setbox0=\hbox{#1}\rlap{%
\kern.5\wd0\dimen0=\ht0\advance\dimen0by-\dp0%\advance\dimen0by+1ex%
\raise.5\dimen0\hbox{\special{x:gsave}\special{x:rotate 90}}}%
\box0\special{x:grestore}}
[...]
\XeTeXmathchardef\bracelu = 0 3 `\⎧
\XeTeXmathchardef\bracemu = 0 3 `\⎨
\XeTeXmathchardef\braceru = 0 3 `\⎩
\XeTeXmathchardef\bracebar = 0 3 `\⎪
\XeTeXmathchardef\braceld = 0 3 `\⎫
\XeTeXmathchardef\bracemd = 0 3 `\⎬
\XeTeXmathchardef\bracerd = 0 3 `\⎭
\def\upbracefill{\rotatebrace{\midshift{$\bracelu$}}%
\cleaders\hbox{\rotatebrace{\midshift{$\bracebar$}}}\hfill
\rotatebrace{\midshift{$\bracemu$}}%
\cleaders\hbox{\rotatebrace{\midshift{$\bracebar$}}}\hfill
\rotatebrace{\midshift{$\braceru$}}}%
\def\downbracefill{\rotatebrace{\midshift{$\braceld$}}%
\cleaders\hbox{\rotatebrace{\midshift{$\bracebar$}}}\hfill
\rotatebrace{\midshift{$\bracemd$}}%
\cleaders\hbox{\rotatebrace{\midshift{$\bracebar$}}}\hfill
\rotatebrace{\midshift{$\bracerd$}}}%
Where Cambria Math is loaded for the 3. family.
However, the height of these braces are bigger because of the middle part of the brace, and I can't find a way to shrink/limit the height of the box, without which the brace would fall apart.
I originally used the :vertical parameter for the font, which I figured has no dimension in horizontal text.
So how can I make this horizontal brace height smaller? (preferably to the character width, which is now the height.)
The example above uses Calibri for normal text and Cambria for mathematical equations, (just as Office 2007 does) as well as for the braces, as in a Hungarian notes of a class of mine.
\Umathaccent,\Umathbotaccent,\Umathaccentsand more, check ntg.nl/maps/38/04.pdf. You can useluaotfloadpackage with plain TeX, and most of your code can be reused, but you get better quality OpenType math. – Khaled Hosny Sep 26 '10 at 18:06:)– Will Robertson Oct 5 '10 at 10:48