How do you define your environment such as to use […] for some parameter(s)?

Question

In an environment, say

\newenvironment{whatever}[3]{...}{...}

you instantiante it (if that's the word for it in LaTeX) by

\begin{whatever}{first}{second}{third}
   ...
\end{whatever}

But how do you define an environment such that you instantiate it by

\begin{whatever}[some_optional]{first}{second}{third}
   ...
\end{whatever}

I'm not talking about default values for the parameters first, second or third, but an entirely optional parameter, such as in

\begin{figure}[htb]
    ...         ^--- THIS!
\end{figure}

Answer 1

\newenvironment{whatever}[4][??]
  {...}
  {...}

\begin{whatever}{one}{two}{three}

is #1= {??}
   #2= one
   #3= two
   #4= three

\begin{whatever}[zero]{one}{two}{three}

is #1= zero
   #2= one
   #3= two
   #4= three

\newenvironment{whatever}[4][foo]
  {...}
  {...}

\begin{whatever}{one}{two}{three}

is #1= foo % preset to foo, if missing
   #2= one
   #3= two
   #4= three

\begin{whatever}[bar]{one}{two}{three}

is #1= bar   % the optional argument 
   #2= one
   #3= two
   #4= three

Answer 2

The xparse package provides an easy user interface to specify optional arguments to environments (in various orders) including commands/macros. For this you define your environment using \NewDocumentEnvironment, while regular macros use \NewDocumentCommand. The former has the following syntax:

\NewDocumentEnvironment{<env>}{<arg spec>}{<beg env>}{<end env>}

<env>, <beg env> and <end env> act in the traditional way to define the environment <env> that executes code <beg env> when using \begin{<env>} (or \<env>) and executes <end env> when using \end{<env>} (or \end<env>). <arg spec> contains a sequence of argument specifications to the environment that can intermix optional and mandatory arguments as needed. In short, this is how they are specified (omitting some options; for more details, read the xparse package documentation):

• m - a mandatory argument (requires { })
• o - an optional argument (uses [ ])
• O{<default>} - an optional argument similar to o but returns <default> if the argument is not given (uses [ ])
• d<token1><token2> - an optional argument with special delimiters (uses <token1> <token2>)
• D<token1><token2>{<default>} - an optional argument similar to d but returns <default> if the argument is not given (uses <token1> <token2>)

Here are some examples:

\documentclass{article}
\usepackage{xparse}% http://ctan.org/pkg/xparse
\begin{document}
\NewDocumentEnvironment{whatever}{m o O{foobar} m}{}{}
\NewDocumentEnvironment{whomever}{O{yes} D(){no}}{}{}

% #1=foo; #2=\NoValue; #3=foobar; #4=bar
\begin{whatever}{foo}{bar} foo \end{whatever}

% #1=foo; #2=bar; #3=foobar; #4=bar
\begin{whatever}{foo}[bar]{bar} foo \end{whatever}

% #1=yes; #2=no
\begin{whomever} foo \end{whomever}

% #1=no; #2=yes
\begin{whomever}[no](yes) foo \end{whomever}

\end{document}

As is obvious, this interface extends the existing "single optional argument" specification to a much wider range.