Classically (La)TeX has no knowledge of the physical paper size, it just sets up a text block area of a give width and height. The "page margins" were just an artifact of where the dvi driver positioned that text block on the page.
PDF and pdftex change that a bit as pdftex is essentially its own driver so needs to be concerned with that aspect and the pdf format itself has a notion of the intrinsic size of each page, which is used by pdf viewers to clip the view or size the window.
The latex format mostly uses
\textheight (the area of the text block) but LaTeX2e introduces
\paperwidth which are typically set by options such as
a4paper. They are not used much although the standard classes use them in some initial calculations to set
The graphics package uses several different back ends (
pdftex, etc) and if it detects that it is running under pdftex then it sets the pdftex primitives
\pdfpageheight to the values of
\paperheight respectively) This has the effect of specifying latex's idea of the physical page size in the pdf output and the viewer uses that as you noticed. (Several other packages now do the same, notably
geometry). This isn't strictly to do with graphics inclusion but the idea back then was that all driver-specific code would be in one customisable file
pdftex.def, ... and this same file is loaded by
color and any other package that needed back end driver support, so it seemed a natural place to add this setting.
For reasons to do with historical compatibility, the default paper size in LaTeX is US letter, however the default page size in your pdf viewer is probably A4 if you are not in the USA. This means that the default behaviour is to typeset the document for US letter but display it in a viewer clipped to A4, which sort of works but is sub-optimal.
The effect of the graphics package making things match is that you noticed:-) Especially as it made them align the wrong way, as latex's default page size was communicated to the pdf browser. By adding
[a4paper] you not only tell the pdf browser that you want that size, so it doesn't clip the bottom, latex actually formats the document for A4 rather than US Letter, to match the physical page dimensions.