The following .lyx file works for me. I went to Document > Settings > Fonts and clicked on "use non-TeX fonts", then I set the fonts to the Free{Serif,Sans,Mono} fonts. Then I exported to PDF with XeTeX.
#LyX 2.1 created this file. For more info see http://www.lyx.org/
\lyxformat 413
\begin_document
\begin_header
\textclass article
\use_default_options true
\maintain_unincluded_children false
\language hebrew
\language_package default
\inputencoding auto
\fontencoding global
\font_roman FreeSerif
\font_sans FreeSans
\font_typewriter FreeMono
\font_default_family default
\use_non_tex_fonts true
\font_sc false
\font_osf false
\font_sf_scale 100
\font_tt_scale 100
\graphics default
\default_output_format default
\output_sync 0
\bibtex_command default
\index_command default
\paperfontsize default
\spacing single
\use_hyperref false
\papersize default
\use_geometry false
\use_amsmath 1
\use_esint 1
\use_mathdots 1
\use_mhchem 1
\cite_engine basic
\use_bibtopic false
\use_indices false
\paperorientation portrait
\suppress_date false
\use_refstyle 1
\index Index
\shortcut idx
\color #008000
\end_index
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\paragraph_indentation default
\quotes_language english
\papercolumns 1
\papersides 1
\paperpagestyle default
\tracking_changes false
\output_changes false
\html_math_output 0
\html_css_as_file 0
\html_be_strict false
\end_header
\begin_body
\begin_layout Standard
\begin_inset Formula $a+b=\text{םולש}$
\end_inset
\end_layout
\end_body
\end_document
The following .tex files is produced from the above .lyx file:
%% LyX 2.1.0dev created this file. For more info, see http://www.lyx.org/.
%% Do not edit unless you really know what you are doing.
\documentclass[english,hebrew]{article}
\usepackage{amstext}
\usepackage{fontspec}
\setmainfont[Mapping=tex-text]{FreeSerif}
\setsansfont[Mapping=tex-text]{FreeSans}
\setmonofont{FreeMono}
\usepackage{xunicode}
\usepackage{polyglossia}
\setdefaultlanguage{hebrew}
\setotherlanguage{english}
\begin{document}
$a+b=\text{םולש}$
\end{document}
I get the following output:

\text{\hebgimel}
to temporarily "release" you from math mode constraints while remaining in math mode for the main expression (requiresamsmath
)..lyx
file and the exported.tex
file please? Does Hebrew work for you outside of math formulas? Are you using XeTeX to compile?