The border of the rectangle is drawn because you are using the
\draw command, which, by definition, draws the specified path (here the rectangle). The fact that you are adding the option
fill=orange just tells TikZ that in addition it should fill the path with orange. Replacing
\fill will do what you are expecting.
\tikz \fill [orange] (0.1,0.1) rectangle (0.2,0.2);
As noted in the comments, another option is to use
\draw [fill=orange,draw=none]. This amounts to telling TikZ to draw the path with an invisible color and fill it with orange, hence the result is the same. As drawing the path with an invisible color is maybe a little strange, it is perhaps more natural to use instead the general
\path [fill=orange]. Note that
\draw is simply a shortcut for
\path [draw] and in a similar manner,
\fill is a shortcut for
\path [fill]. Similarly, you have the command
\filldraw which is an abbreviation for
\path [fill,draw]. In full generality, colors to use for either filling or drawing can be specified by replacing
draw=color in the options.
All these commands are documented in section 15 of the TikZ manual (and especially 15.3 and 15.4), with details and examples (accessible by the command
texdoc tikz in a shell or here).