The following .lyx file works for me. I went to Document > Settings > Fonts and clicked on "use non-TeX fonts", then I set the fonts to the Free{Serif,Sans,Mono} fonts. Then I exported to PDF with XeTeX.
#LyX 2.1 created this file. For more info see http://www.lyx.org/
\lyxformat 413
\begin_document
\begin_header
\textclass article
\use_default_options true
\maintain_unincluded_children false
\language hebrew
\language_package default
\inputencoding auto
\fontencoding global
\font_roman FreeSerif
\font_sans FreeSans
\font_typewriter FreeMono
\font_default_family default
\use_non_tex_fonts true
\font_sc false
\font_osf false
\font_sf_scale 100
\font_tt_scale 100
\graphics default
\default_output_format default
\output_sync 0
\bibtex_command default
\index_command default
\paperfontsize default
\spacing single
\use_hyperref false
\papersize default
\use_geometry false
\use_amsmath 1
\use_esint 1
\use_mathdots 1
\use_mhchem 1
\cite_engine basic
\use_bibtopic false
\use_indices false
\paperorientation portrait
\suppress_date false
\use_refstyle 1
\index Index
\shortcut idx
\color #008000
\end_index
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\paragraph_indentation default
\quotes_language english
\papercolumns 1
\papersides 1
\paperpagestyle default
\tracking_changes false
\output_changes false
\html_math_output 0
\html_css_as_file 0
\html_be_strict false
\end_header
\begin_body
\begin_layout Standard
\begin_inset Formula $a+b=\text{םולש}$
\end_inset
\end_layout
\end_body
\end_document
The following .tex files is produced from the above .lyx file:
%% LyX 2.1.0dev created this file. For more info, see http://www.lyx.org/.
%% Do not edit unless you really know what you are doing.
\documentclass[english,hebrew]{article}
\usepackage{amstext}
\usepackage{fontspec}
\setmainfont[Mapping=tex-text]{FreeSerif}
\setsansfont[Mapping=tex-text]{FreeSans}
\setmonofont{FreeMono}
\usepackage{xunicode}
\usepackage{polyglossia}
\setdefaultlanguage{hebrew}
\setotherlanguage{english}
\begin{document}
$a+b=\text{םולש}$
\end{document}
I get the following output:

\text{\hebgimel}
to temporarily "release" you from math mode constraints while remaining in math mode for the main expression (requiresamsmath
). – barbara beeton Dec 8 '13 at 15:18.lyx
file and the exported.tex
file please? Does Hebrew work for you outside of math formulas? Are you using XeTeX to compile? – scottkosty Dec 9 '13 at 4:32