Inserting _any_ unicode character in Latex document

Question

I am totally new to XeLaTex and LuaLaTex, but I have read that both allow to write LaTeX documents in UTF-8.

That actually works for some unicode characters, but not for all.

See the following example document:

\documentclass{scrartcl}

\usepackage{fontspec}

\setmainfont{Latin Modern Sans}

\begin{document}
A line with normal spaces.

A\,line\,with\,latex\,thin\,spaces.

A line with utf-8 thin spaces.

ÄÆÿ⇒☑ßдάრ你ん녕.
\end{document}

XeLaTeX generates the following:

LuaLaTex generates the following:

In the logfiles I see the following lines:

XeLaTeX:

Missing character: There is no   in font Latin Modern Sans 10 Regular/OT:script
=latn;language=DFLT;mapping=tex-text;!
Missing character: There is no   in font Latin Modern Sans 10 Regular/OT:script
=latn;language=DFLT;mapping=tex-text;!
Missing character: There is no   in font Latin Modern Sans 10 Regular/OT:script
=latn;language=DFLT;mapping=tex-text;!
Missing character: There is no   in font Latin Modern Sans 10 Regular/OT:script
=latn;language=DFLT;mapping=tex-text;!
Missing character: There is no   in font Latin Modern Sans 10 Regular/OT:script
=latn;language=DFLT;mapping=tex-text;!
Missing character: There is no ⇒ in font Latin Modern Sans 10 Regular/OT:script
=latn;language=DFLT;mapping=tex-text;!
Missing character: There is no ☑ in font Latin Modern Sans 10 Regular/OT:script
=latn;language=DFLT;mapping=tex-text;!
Missing character: There is no д in font Latin Modern Sans 10 Regular/OT:script
=latn;language=DFLT;mapping=tex-text;!
Missing character: There is no ά in font Latin Modern Sans 10 Regular/OT:script
=latn;language=DFLT;mapping=tex-text;!
Missing character: There is no რ in font Latin Modern Sans 10 Regular/OT:script
=latn;language=DFLT;mapping=tex-text;!
Missing character: There is no 你 in font Latin Modern Sans 10 Regular/OT:script
=latn;language=DFLT;mapping=tex-text;!
Missing character: There is no ん in font Latin Modern Sans 10 Regular/OT:script
=latn;language=DFLT;mapping=tex-text;!
Missing character: There is no 녕 in font Latin Modern Sans 10 Regular/OT:script
=latn;language=DFLT;mapping=tex-text;!

LuaLaTeX:

Missing character: There is no   (U+202F) in font LatinModernSans:mode=node;sc
ript=latn;language=DFLT;+tlig;!
Missing character: There is no   (U+202F) in font LatinModernSans:mode=node;sc
ript=latn;language=DFLT;+tlig;!
Missing character: There is no   (U+202F) in font LatinModernSans:mode=node;sc
ript=latn;language=DFLT;+tlig;!
Missing character: There is no   (U+202F) in font LatinModernSans:mode=node;sc
ript=latn;language=DFLT;+tlig;!
Missing character: There is no   (U+202F) in font LatinModernSans:mode=node;sc
ript=latn;language=DFLT;+tlig;!
Missing character: There is no ⇒ (U+21D2) in font LatinModernSans:mode=node;sc
ript=latn;language=DFLT;+tlig;!
Missing character: There is no ☑ (U+2611) in font LatinModernSans:mode=node;sc
ript=latn;language=DFLT;+tlig;!
Missing character: There is no д (U+0434) in font LatinModernSans:mode=node;scr
ipt=latn;language=DFLT;+tlig;!
Missing character: There is no ά (U+03AC) in font LatinModernSans:mode=node;scr
ipt=latn;language=DFLT;+tlig;!
Missing character: There is no რ (U+10E0) in font LatinModernSans:mode=node;sc
ript=latn;language=DFLT;+tlig;!
Missing character: There is no 你 (U+4F60) in font LatinModernSans:mode=node;sc
ript=latn;language=DFLT;+tlig;!
Missing character: There is no ん (U+3093) in font LatinModernSans:mode=node;sc
ript=latn;language=DFLT;+tlig;!
Missing character: There is no 녕 (U+B155) in font LatinModernSans:mode=node;sc
ript=latn;language=DFLT;+tlig;!

So, both are missing the same characters. XeLaTeX seems to replace them by spaces, whereas LuaLaTex just leaves them out.

Interestingly, XeLaTeX seems to actually correctly set the narrow non-breaking spaces (the third output line) even though it still complains about them in the log file.

Now, when I enter those very same input lines into LibreOffice, all these characters are correctly displayed in the same font:

So it seems that this font actually does provide those characters. And the same problem applies to other fonts I have tried, so it doesn't seem to be a font problem.

Now, what is wrong here?

What do I need to do to have arbitrary unicode characters int my LaTeX documents correctly set in the output?

Update

As the comments point out, it is actually a font problem. Latin Modern doesn't include all the glyphs. Google Noto does include some (like the narrow space), but not all (e.g. ⇒ and ☑ are still missing).

@LaTeXer metioned "fallback fonts". That is actually what I need here. After some research I found this question and somewhere a reference to the ucharclasses package.

ucharclasses seems like a "clean" approach, but has the drawback, that I have to specify all the transitions. It doesn't allow me to say "if a character is not found in the main font, try to find it in the next fallback font". I would prefer using the glyphs from my main font if available, because the would look more consistent with the remaining glyphs. Also I would need to find out which unicode characters I actually use and which blocks they are defined in.

The above mentioned stackexchange link describes exactly, what I need. But unfortunately there doesn't seem to be an easy solution. This answer attracts me, but seems quite hacky.

So, what is your recommended approach to handle such "font fallback"?

Update 2

As the narrow space seems to not be available in Latin Modern, what does LaTeX use when it processes a \, sequence? Does it calculate the needed space by itself? I assume that could be different from a possible glyph contained in the font. Would it be better to rely on LaTeX to calculate such narrow spaces or is a font providing that space (like Google Noto) the better choice?

Update 3

As both XeLaTeX and LuaLaTeX fail silently and only mention the missing glyphs in the logfile, is there an option to have them emit a warning or even an error when glypths are missing, so that I can be aware of that problem?

Answer 1

In LuaLaTeX this can be done using the combofont package by Ulrike Fischer. This allows to specify fallback fonts, and fallback fonts for the fallback fonts.

MWE, mostly taken from the combofont manual:

\documentclass{scrartcl}
\usepackage{combofont}
\usepackage{fontspec}
\setmainfont{Latin Modern Sans}
\setupcombofont{multiscript-regular}
 {
  {file:lmsans10-regular.otf:\combodefaultfeat} at #1pt,
  {file:DejaVuSans.ttf} at #1pt,
  {file:NotoSansCJK-Regular.ttc(0)} at #1pt
 }
 {
   {} ,
   fallback,
   fallback
 }
\DeclareFontFamily{TU}{multiscript}{}
\DeclareFontShape {TU}{multiscript}{m}{n} {<->combo*multiscript-regular}{}
\begin{document}
A line with normal spaces.

A\,line\,with\,latex\,thin\,spaces.

A line with utf-8 thin spaces.

ÄÆÿ⇒☑ßдάრ你ん녕.

\rule{\textwidth}{1pt}

\fontfamily{multiscript}\selectfont
A line with utf-8 thin spaces.

ÄÆÿ⇒☑ßдάრ你ん녕.
\end{document}

Result:

Note that you need to specify which fonts you want to use as fallback. Also, Noto Sans CJK is a .ttc font, i.e., a TrueType Collection, for which you need to specify the index in the collection. JP-Regular is the first font in the collection, so index 0. You can find these indexes in different ways, for example by looking at the output of fc-list.

---

About the thin space: \, is defined as

\relax \ifmmode \mskip \thinmuskip \else \thinspace \fi

So, if in math mode, then \mskip, if in text mode, \thinspace. And \thinspace is defined as \kern .16667em, which is conditional on the current font size (1em is the size of a 'normal' character, which changes with the font size, and is defined in the font file). If the fonts are designed sensibly the result will most likely be fine when using \,.

---

About the missing character warnings: you can set \tracinglostchars=2 in your preamble to make the warnings show in the terminal output and not just in the logfile (from Getting xetex to complain when a character is missing).