I hope you are aware of the fact that your brace (the decorated path) is only one millimeter long. That’s tiny!
Nevertheless, you can decrease the amplitude
to get a better result, i.e.
decoration={brace, mirror, amplitude=+.6pt}
Code
\documentclass[tikz]{standalone}
\usetikzlibrary{decorations.pathreplacing}
\begin{document}
\foreach \lw in {ultra thin, very thin, thin}{% linewidth: .1pt, .2pt, .4pt
\begin{tikzpicture}[\lw]
\draw[decoration={brace,mirror}] \foreach \amp in {0,.1,.2,.3,.4,.5,.6,.7,.8,.9,1} {
[/pgf/decoration/amplitude=\amp pt] (\amp,0) decorate {-- ++(up:.1)}
node [above,scale=.4] {\tiny\amp}};
\draw[ultra thin] (-.1,0) -- + (left:.02) |- +(up:.1)
node[scale=.4, left, pos=.25, inner sep=+1pt] {\tiny1\,mm};
\end{tikzpicture}}
\end{document}
Output
