# Why won't this compile? Argument of \h has an extra {

\documentclass{article}
\usepackage{amsmath}

\begin{document}
\newcommand{\x}[1][1]{\ensuremath{a_{n-#1}}}
\newcommand{\e}[1][\left(1+\dfrac{1+\sqrt{5}}{2}\right)-\left(\dfrac{1-\sqrt{5}}{2}\right)^{n-2}\left(1+\dfrac{1-\sqrt{5}}{2}\right)]{\ensuremath{\\\\=\dfrac{\left(\dfrac{1+\sqrt{5}}{2}\right) ^{n-2}#1}{\sqrt{5}}}}
\newcommand{\f}{\ensuremath{\left(\dfrac{1-\sqrt{5}}{2}\right)^{n-2}}}
\newcommand{\g}[1][n-1]{\ensuremath{\left(\dfrac{1-\sqrt{5}}{2}\right)^{#1}}}
\newcommand{\h}[1][n-1]{\ensuremath{\left(\dfrac{1+\sqrt{5}}{2}\right)^{#1}}}
\newcommand{\n}{\ensuremath{\left(\dfrac{1+\sqrt{5}}{2}\right) ^{n-2}}}

$a_n=\x+\x[2]\e[-\f+\h\g]\e\e[\h[2]]$

\end{document}


use

\e[{\h[n-1]}]


Put the inner \h[..] into braces

If you define commands with optional argument through xparse, then you can nest them.

\documentclass{article}
\usepackage{amsmath}
\usepackage{xparse}

\newcommand{\grplus}{\dfrac{1+\sqrt{5}}{2}}
\newcommand{\grminus}{\dfrac{1-\sqrt{5}}{2}}

\newcommand{\eopt}{%
\left(1+\grplus\right)-\f\left(1+\grminus\right)%
}

\NewDocumentCommand{\x}{O{1}}{a_{n-#1}}
\NewDocumentCommand{\e}{O{\eopt}}{\\&=\dfrac{\n #1}{\sqrt{5}}}

\NewDocumentCommand{\g}{O{n-1}}{\left(\grminus\right)^{#1}}
\NewDocumentCommand{\h}{O{n-1}}{\left(\grplus\right)^{#1}}

\newcommand{\n}{\g[n-2]}
\newcommand{\f}{\h[n-2]}

\begin{document}

\begin{aligned}[t] a_n&=\x+\x[2]\e[-\f+\h\g]\e\e[\h[2]] \end{aligned}


I also used aligned in order to get a better alignment.

There's no reason for plaguing you code with useless \ensuremath.