# Output Devanagari (Hindi) from raw unicode using luatex

I can get the following code to compile, using luatex, with the Hindi/Devanagari characters correctly printed in the pdf:

\documentclass{article}
\usepackage{fontspec}
\setmainfont{Times New Roman}
\newfontscript{Devanagari}{deva,dev2}
\newfontface{\hindi}[Script=Devanagari]{Lohit-Devanagari.ttf}

\begin{document}
Here is normal text.
{\hindi नमस्ते }
\end{document}


However, I'm using a program that outputs the tex and that won't allow me to type the Hindi script into my tex editor; instead, it will only give me the unicode version of the word, "नमस्ते", which is "<U+0928><U+092E><U+0938><U+094D><U+0924><U+0947>".

How can I get luatex to compile correctly from these raw code characters? What I want to compile (to produce a pdf with the single word "नमस्ते") is something like this:

\documentclass{article}
\usepackage{fontspec}
\setmainfont{Times New Roman}
\newfontscript{Devanagari}{deva,dev2}
\newfontface{\hindi}[Script=Devanagari]{Lohit-Devanagari.ttf}

\begin{document}
Here is normal text.
{\hindi <U+0928><U+092E><U+0938><U+094D><U+0924><U+0947> }
\end{document}


...but that won't work.

• Can you get your program to output \char"0928\char"092E\char"0938\char"094D\char"0924 \char"0947 instead of <U+0928><U+092E><U+0938><U+094D><U+0924><U+0947>? – Mico Apr 20 at 3:31
• Yes, I could do that! What would the full script then need to look like? – lethalSinger Apr 20 at 3:32

[(i) Added an extra operation in the Lua function 'conv' to address the OP's follow-up request. (ii) Implemented Ulrike Fischer's suggestion to use ^^^^ notation to typeset 4-byte characters. ]

Since you're using LuaLaTeX, here's a solution that employs a Lua function to convert strings of the form '<U%+(.-)>' to '^^^^%1'; here, %+ represents the literal character + and %1 represents the capture of the non-greedy pattern (.-)> -- in words: "0 or more characters of any kind up to, but not including, >". The Lua function takes care to append {} to each converted number; this is important if the input string contains whitespace.

In addition, the code also sets up a LaTeX macro that acts as a front-end for the Lua function. Thus, one may call the Lua function via a \conv{<your string here>} directive.

You can either manually encase the sequences of unicode code in \conv{...} statements or, depending on how far you can get your scripting program to do the work for you, instruct the program to encase the sequences of unicode code in \conv{...} statements automatically.

\documentclass{article}
\usepackage{fontspec}
\setmainfont{Times New Roman}
\newfontscript{Devanagari}{deva,dev2}
\newfontface{\hindi}[Script=Devanagari]{Lohit-Devanagari.ttf}

%%%% -- copy the next eight lines of code to your document --
\usepackage{luacode} % for 'luacode' env. and '\luastringN' macro
\begin{luacode}

function conv ( s )
s = s:gsub ( '<U%+(.-)>' , function ( x )
return '^^^^'..string.lower(x)..'{}'
end )
tex.sprint ( s )
end

\end{luacode}
\newcommand\conv[1]{\directlua{conv(\luastringN{#1})}}

\begin{document}
Latin-alphabet text.

{\hindi नमस्ते }

{\hindi \conv{<U+0928><U+092E><U+0938><U+094D><U+0924><U+0947>} }

{\hindi \conv{<U+0928><U+092E><U+0938><U+094D><U+0924><U+0947> <U+0930><U+093E><U+091C>}}
\end{document}

• This gets incredibly close. The only problem now is with breaks between words, which get ignored. E.g. "नमस्ते राज" (2 words) gets printed as "नमस्तेराज" (1 single word) even though there is the proper space between the unicode characters: "<U+0928><U+092E><U+0938><U+094D><U+0924><U+0947> <U+0930><U+093E><U+091C>". How can I fix the spacing issue? – lethalSinger Apr 20 at 4:06
• @lethalSinger - Please see the updated answer I just posted. (The solution is to add a second gsub (short for "global substitution") operation.) – Mico Apr 20 at 5:24
• I wouldn't convert this to \char commands, but to the ^^^^ input notation. \char will not work well e.g. with hyperref and bookmarks. Try out the difference with \section{\char102\char105 ^^66^^69}. – Ulrike Fischer Apr 20 at 9:07
• @UlrikeFischer - Done. :-) A mild complication in implementing your idea was the fact that ^^^^ notation expects hexadecimal number to be in lowercase. E.g., ^^^^092E throws an error, whereas ^^^^092e does not. – Mico Apr 20 at 11:17