Adding fontspec looks to be enough, but if it still doesn't work, then you can try adding the missing characters manually as well.
(Note that on MacTeX I had to use hungarian
instead of magyar
in the babel settings, as only the former is supported)
\documentclass{article}
\usepackage{fontspec, newunicodechar}
\usepackage[hungarian]{babel}
\usepackage{lmodern}
\newunicodechar{ű}{\H{u}}
\newunicodechar{ő}{\H{o}}
\begin{document}
árvíztűrő tükörfúrógép
ÁRVÍZTŰRŐ TÜKÖRFÚRÓGÉP
\end{document}
There are still some LaTeX warnings displayed though:
LaTeX Warning: Please use \usepackage{t1enc} with
\usepackage[hungarian]{babel}, to get accented chars hyphenated.
LaTeX Warning: Please use \usepackage[latin2]{inputenc} with
\usepackage[hungarian]{babel}.
But the output is okay:

ű = \H{u}
gives ű=ű. So what does the log file say?