# Lulatex - Unicode character error

I'm using a \LaTeX document using both Devnagari and Latin fonts.

The command used is: lualatex -shell-escape -interaction nonstopmode

However, despite selecting the appropriate fonts. The minimal code is:

I also tried using the polyglossia package, (as per https://tex.stackexchange.com/a/24731/50653), but still no difference.

\documentclass[10pt]{article}

\usepackage{polyglossia}
\setdefaultlanguage{english}
\setotherlanguage{hindi}

% From https://tex.stackexchange.com/a/37251
\usepackage{fontspec}
\setmainfont{Noto Serif} % sets the roman font
\setsansfont{Noto Sans} % sets the sans font
\setmonofont{Fira Code} % sets the monospace font
\defaultfontfeatures{Ligatures=TeX} % makes this a feature for all selected fonts

\newfontfamily\hindifont[Script=Devanagari]{Noto Serif Devanagari}
\newfontfamily\hindifontsf[Script=Devanagari]{Noto Sans Devanagari}

\begin{document}

All the places marked with red triangles are \emph{sites} from which archaeologists have found evidence of hunter-gatherers. (Hunter-gatherers lived in many more places. Only some are shown on the map). Many sites were located near sources of water, such as rivers and lakes.
येह एक प्रयोग है हिन्दी की लिखावट देखने के लिए।
\end{document}


The obtained result is:

While one expects the Devanagari characters to be rendered as well.

I had earlier also tried

\documentclass[10pt]{article}

\usepackage[hindi, english]{babel}

% From https://tex.stackexchange.com/a/37251
\usepackage{fontspec}
\setmainfont{Noto Serif} % sets the roman font
\setsansfont{Noto Sans} % sets the sans font
\setmonofont{Fira Code} % sets the monospace font
\defaultfontfeatures{Ligatures=TeX} % makes this a feature for all selected fonts

\begin{document}

All the places marked with red triangles are \emph{sites} from which archaeologists have found evidence of hunter-gatherers. (Hunter-gatherers lived in many more places. Only some are shown on the map). Many sites were located near sources of water, such as rivers and lakes.
येह एक प्रयोग है हिन्दी की लिखावट देखने के लिए।
\end{document}

• Hi, welcome to TeX.SE. Can you please complete your sample code? Currently it is missing the documentclass line... – Thruston Sep 23 at 12:40
• And can you tell us which TeX engine you are using? If you are using lualatex then you don't need the old pdflatex packages (fontenc, inputenv, textcomp).... – Thruston Sep 23 at 12:41
• Thanks; edits made. – Rohan Sep 23 at 12:58

## 2 Answers

There are two issues at play here: The primary problem is that the selected font does not contain the characters in question, the other issue is that using fontspec fontloading commands without explicitly specifying a script and language default to Latin script and therefore does not shape indic scripts correctly.

If you do not want to worry about the right script/language parameters for fontspec, you can use babel's fontspec wrapper \babelfont to specify the fonts for your languages. For this to work correctly (especially for language specific shaping to work) you have to indicate language changes in your document using \selectlanguage, \foreignlanguage or the otherlanguage / otherlanguage* environments. If you write a whole paragraph in another language you can use \selectlanguage{new-language}, for parts of a paragraph there is \foreignlanguage{new-language}{text-in-new-language}.

You can find details about all this in the babel manual, section 1.2 (Multilingual documents), section 1.7&1.8 (Basic/Auxiliary language selectors) and section 1.14 (Selecting fonts).

\documentclass[10pt]{article}

\usepackage[hindi, english]{babel}

\babelfont{rm}{Noto Serif} % sets the roman font
\babelfont{sf}{Noto Sans} % sets the sans font
\babelfont{tt}{Fira Code} % sets the monospace font
\babelfont[hindi]{rm}{Noto Serif Devanagari}
\babelfont[hindi]{sf}{Noto Sans Devanagari}

\begin{document}

All the places marked with red triangles are \emph{sites} from which archaeologists have found evidence of hunter-gatherers. (Hunter-gatherers lived in many more places. Only some are shown on the map). Many sites were located near sources of water, such as rivers and lakes.
\foreignlanguage{hindi}{येह एक प्रयोग है हिन्दी की लिखावट देखने के लिए।}
More English.

\selectlanguage{hindi}

येह एक प्रयोग है हिन्दी की लिखावट देखने के लिए। येह एक प्रयोग है हिन्दी की लिखावट देखने के लिए।
येह एक प्रयोग है हिन्दी की लिखावट देखने के लिए। येह एक प्रयोग है हिन्दी की लिखावट देखने के लिए।
येह एक प्रयोग है हिन्दी की लिखावट देखने के लिए। येह एक प्रयोग है हिन्दी की लिखावट देखने के लिए।

\selectlanguage{english}

And English again.
\end{document}


As Davislor mentioned in a comment, the following requires a pretty modern TeX installation (updated in February 2020 or later) and therefore might not work on your system or break if you try to compile your document on another system:

If you really, really can't insert such language switching commands, you only use one language from every script system and you do not care about proper paragraph spacing in Devanagari text (e.g. because it is mostly latin) you can also use onchar=ids fonts: Replace

\usepackage[hindi, english]{babel}


with

\usepackage[english]{babel}
\babelprovide[import, onchar=ids fonts]{hindi}


Then babel automatically switches the font and hyphenation patterns to Hindi when it encounters Devanagari characters. (Of course, you can switch english and hindi if you mostly have Hindi text.)

• Don't forget otherlanguage* and \foreignlanguage that, in this case, are better. – egreg Sep 23 at 13:35
• Thanks. But unfortunately, I cannot declare the languages for each instance as the language switch happens frequently, and also because I'm using Org-mode (Emacs) I've tried using the polyglossia package (and updated the question), as per examples found, but still no luck – Rohan Sep 23 at 13:46
• You can also enter \babeltag{hindi=hindi} in Babel to provide the commands \texthindi, \hindi and \begin{hindi}, like Polyglossia. – Davislor Sep 23 at 14:17
• @Davislor I just looked it up again, it is available in a fully updated TeXLive 2019, but it had been added very late (~January 2020). So it might not be available in many fixed TeXLive 2019 versions. – Marcel Krüger Sep 23 at 15:03
• @Rohan \babelfont[hindi]{rm}[Renderer=HarfBuzz]{...} (I think, otherwise you need [Renderer=...] at the end of the line) – Marcel Krüger Sep 23 at 15:22

Welcome to TeX.SX!

The LuaTeX shaper dev2 has been a bit troublesome with Indic scripts, but there's been another version called deva out, as mentioned in https://tex.stackexchange.com/a/362738/90407. However, recently, LuaTeX has additionally support to use the HarfBuzz shaper instead of the built-in one. Both variants should give comparable results, albeit with slight differences. In both cases we need to create font switches for the new font family however. Using polyglossia we can add another language and define new font families for that language which will handle all the font switching between sans-serif and serif automatically:

### LuaTeX deva

\documentclass{article}

\usepackage{polyglossia}
\setdefaultlanguage{english}
\setotherlanguage{hindi}

\usepackage{fontspec}
% define Devanagari script using deva' backend
\newfontscript{Devanagari}{deva,dev2}

\setmainfont{Noto Serif}
\setsansfont{Noto Sans}
% Create font switches for Devanagari font
% These are hard-coded into polyglossia, if we name them differently
% we get:
%     Package polyglossia Error: The current latin font [...] does
%     not contain the "Devanagari" script!
%     Please define \devanagarifont with \newfontfamily command.
\newfontfamily{\devanagarifont}{Noto Serif Devanagari}[Script=Devanagari]
\newfontfamily{\devanagarifontsf}{Noto Sans Devanagari}[Script=Devanagari]

\defaultfontfeatures{Ligatures=TeX}

\begin{document}

All the places marked with red triangles are \emph{sites} from which archaeologists have found evidence of hunter-gatherers. (Hunter-gatherers lived in many more places. Only some are shown on the map). Many sites were located near sources of water, such as rivers and lakes.
\texthindi{येह एक प्रयोग है हिन्दी की लिखावट देखने के लिए।}

All the places marked with red triangles are \emph{sites} from which archaeologists have found evidence of hunter-gatherers. (Hunter-gatherers lived in many more places. Only some are shown on the map). Many sites were located near sources of water, such as rivers and lakes.
\texthindi{\textsf{येह एक प्रयोग है हिन्दी की लिखावट देखने के लिए।}}
\end{document}


The result is:

### LuaTeX with HarfBuzz (LuaHBTeX)

Alternatively, we don't define a new script but simply ask HarfBuzz to do the heavy lifting:

\documentclass{article}

\usepackage{polyglossia}
\setdefaultlanguage{english}

\usepackage{fontspec}

\setmainfont{Noto Serif}
\setsansfont{Noto Sans}
% Create font switches for Devanagari font
% These are hard-coded into polyglossia, if we name them differently
% we get:
%     Package polyglossia Error: The current latin font [...] does
%     not contain the "Devanagari" script!
%     Please define \devanagarifont with \newfontfamily command.
\newfontfamily{\devanagarifont}{Noto Serif Devanagari}[Script=Devanagari,Renderer=Harfbuzz]
\newfontfamily{\devanagarifontsf}{Noto Sans Devanagari}[Script=Devanagari,Renderer=Harfbuzz]

\defaultfontfeatures{Ligatures=TeX}

\begin{document}

All the places marked with red triangles are \emph{sites} from which archaeologists have found evidence of hunter-gatherers. (Hunter-gatherers lived in many more places. Only some are shown on the map). Many sites were located near sources of water, such as rivers and lakes.
\texthindi{येह एक प्रयोग है हिन्दी की लिखावट देखने के लिए।}

All the places marked with red triangles are \emph{sites} from which archaeologists have found evidence of hunter-gatherers. (Hunter-gatherers lived in many more places. Only some are shown on the map). Many sites were located near sources of water, such as rivers and lakes.
\texthindi{\textsf{येह एक प्रयोग है हिन्दी की लिखावट देखने के लिए।}}
\end{document}


Which yields:

Taking my browsers rendering as reference, this looks quite okay to me.

If you don't care about proper spacing, you can do this without polyglossia, with something like this:

\usepackage[Devanagari]{ucharclasses}

\setTransitionsFor{Devanagari}{\devanagarifont}
`

However, this will only switch the font and do away with any other language adjustments otherwise implemented by polyglossia.

• Thanks. This works nicely, but needed a way to automate it instead of declaring the language each time, as the switch is rather frequent & I'm using conversion from Org-mode – Rohan Sep 23 at 15:15
• @Rohan Hm, that only works for font switching alone, either with ucharclasses or with babel onchar (as shown in the other answer). However this won't do proper spacing, as it's impossible to detect the language reliably from the script (e.g. French and English have different spacing but it's virtually impossible to tell). – ljrk Sep 23 at 15:32
• Thanks, using \babelfont[hindi]{rm}[Renderer=HarfBuzz]{...} as per the comments on the previous answer solved it. – Rohan Sep 23 at 15:34
• @Rohan Great to hear! I'm personally not such a big fan of babel, but that's mostly to taste :) – ljrk Sep 23 at 15:39