When using the key matrix of math nodes (or matrix of nodes), TikZ intends on interpreting everything between the ampersands as the contents of a node. However, we (the users) often want the ability to modify the style of those nodes on a case-by-case basis. In normal usage, this is done via the parameters to the \node command. Also, we like to modify how the matrix lays out its nodes a bit. So what TikZ does is to allow us to specify these things using special syntax that must occur first in the cell. Stuff in vertical lines is added between the \node command and its contents. So you can draw a specific node by writing |[draw]| and you can name it explicitly by writing |(name)|. In addition, the ampersand is actually a command which can take an optional argument and this translates into an extra spacing between the rows (if given on the first row).
So your square brackets are being eaten up as row-skips (and then ignored as they aren't on the first row) and your vertical lines are getting into the machinery of the node command. To protect each you need to ensure that the first character of the cell is not special. Either enclose the whole contents in braces or just put a {} in front.
Here's both versions:
\documentclass[a4paper,12pt]{article}
%\url{http://tex.stackexchange.com/q/68600/86}
\usepackage{tikz}
\usetikzlibrary{matrix}
\begin{document}
\begin{tikzpicture}
\matrix (m) [matrix of math nodes,
nodes in empty cells,nodes={minimum width=5ex,
minimum height=5ex,outer sep=-5pt},
column sep=1ex,row sep=1ex]{
0 & 0 & 0 & \vdots & \\
0 & {[x]} & 0 & k & \\
0 & 0 & 0 & \vdots & \\
0 & 0 & 0 & {}|1| & \\
0 & 0 & {}[] & 0 & \\
-2 & -1 & 0 & & \\};
\end{tikzpicture}
\end{document}
Incidentally, you could use execute at empty cell={\node {0};} to avoid having to type all those zeros:
\documentclass[a4paper,12pt]{article}
%\url{http://tex.stackexchange.com/q/68600/86}
\usepackage{tikz}
\usetikzlibrary{matrix}
\begin{document}
\begin{tikzpicture}
\matrix (m) [matrix of math nodes,
execute at empty cell={\node {0};},nodes={minimum width=5ex,
minimum height=5ex,outer sep=-5pt},
column sep=1ex,row sep=1ex]{
& & & \vdots \\
& {[x]} & & k \\
& & & \vdots \\
& & & {}|1| \\
& & {}[] & \\
-2 & -1 & & {} \\};
\end{tikzpicture}
\end{document}

\documentclassand the packages that you're using as these might all have bearing on the case. – Andrew Stacey Aug 24 '12 at 14:08