TeX gives you a lot of help to find the problem.
With the code as posted the error is
Runaway argument?
! Paragraph ended before \align* was complete.
<to be read again>
\par
l.12
?
which is telling you that the blank line (paragraph end) on line 12 isn't allowed inside align*
.
So if you remove the blank lines and run it again
You get the error
! Missing { inserted.
<to be read again>
\endgroup
l.23 \end{align*}
This is slightly more obscure as for technical reasons in align
such errors are always reported as being on the last line, however it tells you that the {}
matching is wrong, or actually in this case a^}
should have been a^{2}}
then, running it again, the error is
! LaTeX Error: \begin{document} ended by \end{group*}.
See the LaTeX manual or LaTeX Companion for explanation.
Type H <return> for immediate help.
...
l.25 \end{group*}
I have no idea what that is intended to be so simply delete that line.
It then runs without error but with excessive space before the display as there is an empty row as the first line before the first \\
so re-arranging that produces

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\begin{document}
\section{Problem 5}
\begin{align*}
b + 2y &= a\\
\frac{b}{2} + y &= \frac{a}{2}\\
y &= \frac{a-b}{2}\\
2y &= a-b\\
(x+b)^2 + (\frac{b}{2})^2&=r\\
(a-x)^2 + (\frac{a}{2})^2&=r\\
(x+b)^2 + (\frac{a}{2})^2&=(a-x)^2 + (\frac{a}{2})^2\\
x^2 + 2bx + b^2 + \frac{b^2}{4} &=a^2 -2ax + x^2 + \frac{a^2}{4}\\
2bx + \frac{5b^2}{4} &= -2ax + \frac{5a^{2}}{4}\\
\frac{2bx+2ax}{2} &= \frac{\frac{5a^2}{4}}-\frac{{5b^2}{4}}{2}\\
bx + ax &= \frac{5a^2}{8} - \frac{5b^2}{8}
\end{align*}
\end{document}
\\
is end of line not start, so omit the first\\
also you can not have blank lines within the environment.\end{group*}
\\2bx + \frac{5b^2}{4} &= -2ax + \frac{5a^}{4}
Did you perhaps forget to type in the superscript for5a
?