"In most text typefaces, em dashes have no side bearings, which make them appear very close to the words they separate" (James Felici)
The main problem is that, due to the stems in some characters, the dash looks not so close as in other chars. For example:
The dash looks much more separated from the 1 than from the 6. But if you draw the boxes around each char, you can see that it touches both boxes:
You can see now that the problem is in the "1", which has too much white space at its right. This is intentional, so that all digits have the same width. But in another font it could be different.
The problem depends on the font and on each character in the font. Thus, it is a kerning issue. Each font should define an appropiate kerning between each char and the dashes, so that this kind of effects is not noticeable. Unfortunately kerning information is stored in the font, and you cannot (easily) modify it from TeX (see here).
By the way, just in case someone cares, the code used to produce the above figures is the following:
\documentclass[11pt]{amsart}
% Following lines are taken from "The TeXBook", solution to Exercise 11.5, but modified so that
% it shows the character in addition to the box
\def\dolist{\afterassignment\dodolist\let\next= }
\def\dodolist{\ifx\next\endlist \let\next\relax
\else \\\let\next\dolist \fi
\next}
\def\endlist{\endlist}
\def\hidehrule#1#2{\kern-#1%
\hrule height#1 depth#2 \kern-#2 }
\def\hidevrule#1#2{\kern-#1{\dimen0=#1
\advance\dimen0 by#2\vrule width\dimen0}\kern-#2 }
\def\makeblankbox#1#2{\hbox{\lower\dp0\vbox{\hidehrule{#1}{#2}%
\kern-#1 % overlap the rules at the corners
\hbox to \wd0{\hidevrule{#1}{#2}%
\raise\ht0\vbox to #1{}% set the vrule height
\lower\dp0\vtop to #1{}% set the vrule depth
\hfil\hidevrule{#2}{#1}}%
\kern-#1\hidehrule{#2}{#1}}}}
\def\maketypebox{\makeblankbox{0pt}{.1pt}\llap{\box0}} % <-- Added \llap to show the char
\def\makelightbox{\makeblankbox{.1pt}{.1pt}}
\def\\{\if\space\next\ % assume that \next is unexpandable
\else \setbox0=\hbox{\next}\maketypebox\fi}
\def\demobox#1{\setbox0=\hbox{\dolist#1\endlist}%
\leavevmode\copy0\kern-\wd0}%\makelightbox}
% -- End of borrowed code
% This macro is needed to use an endash inside \demobox
% otherwise the -- would be "broken" as -{}-
\def\endash{\char"7B{}}
\begin{document}
61\endash68
\demobox{%
61\endash68
}
\end{document}