traditionally, the table of contents comes first, and an index (or indexes) come last.
i've seen notation lists (1) as a separate index preceding the main index(es), (2) a "chapter" (i.e., not in index form) just before the index(es), (3) a "chapter" preceding the bibliography, or (4) as a separate prefatory chapter. the separate prefatory chapter is more common in pedagogical works than in academic monographs; this parallels the introduction in an article where the author defines the notation.
bibliography is usually the last thing before the index(es) modulo placement of notation list.
lists of figures and tables usually come just after table of contents.
if one is really traditional, a colophon may be placed after the index(es); it should never be more than one page long.