Zdeněk Wagner, a Czech TeXist, who is not having enough points to post an answer at TeX.SX (or he is probably having different kind of problem), yet, is writing his observations:
The problem is not in
Language, these commands are correct, the problem is in font loading. If you do not specify the script explicitly, the Latin script is assumed and thus the non-Latin characters cannot appear properly. What is missing is the script specification, you have to write
Your example is wrong. All three characters, virama, candrabindu and o-matra, are dependent vowels, you cannot combine more than one of them with a consonant. If you type ल् (with a virama), you cannot add a candrabindu. If it is not an end of a word, you can only add another consonant. You can make a conjunct (ligature) with a candrabindu, i.e. ल्लँ but it must never be followed by any matra. In theory, you could add independent ओ but I cannot imagine how you would pronounce it. It looks really weird. You can, of course, nasalize a matra, e.g. ताँ, but if o-matra or ii-matra is to be nasalized, anusvara is used (merely due to graphical appearance).
There is an evident misprint in the textbook, the correct form is ताँल्लोकान्.
Try the following add-on to see which fonts are used by Firefox:
I am adding an example. The ligature
ll is not having its Unicode code in all three fonts, so the result is correct from
xelatex (left picture; all three cases) and once from
lualatex (right picture; the first line).
% run: xelatex mal-chandas.tex
Update: We could use
luaotfload package instead of
fontspec package to get a correct form with
lualatex; I used
script=devanagari in document body then (I typed
tAMMllokAn at Lexilogos):
% run: lualatex mal-siddhanta.tex
% taaMMllokaan at Lexilogos
\font\malfont=Siddhanta:script=devanagari \malfont ताँल्लोकान्