There are at least three different ways one can write such brackets, using ERTs or the LyX GUI.
To be able to write
\left|\right\rangle, you have to use ERTs, which you add with Insert --> TeX code or Ctrl + L. Add an ERT for the
\left|, exit the ERT, for example by hitting the right arrow key, write the term that should be within the delimiters, then finally add another ERT for
braket package mentioned by cgnieder defines eight new commands,
\Set which uses
\right to stretch the delimiters, and similar commands all lowercase, which doesn't have stretchable delimiters. See below for how it looks in LyX and the corresponding output.
To use this in LyX you can do as follows:
Add the package to the document preamble. Under Document --> Properties --> LaTeX preamble write
When in mathmode, add an ERT with Ctrl + L.
Write the command you want to use. The second curly brace will be autocompleted when you type the first (at least in my copy of LyX).
The GUI-version is to click the Insert delimiters-button on the math toolbar and uncheck "Keep matched".