Getting correct searchable text for Devanagari text

Question

Consider this fairly minimal document, which AFAIK is the recommended way of typesetting Devanagari-script Sanskrit-language text:

\documentclass{article}

\usepackage{fontspec}
\usepackage{polyglossia}
\setmainlanguage{sanskrit}
\newfontfamily\devanagarifont[Script=Devanagari]{Chandas}

\begin{document}
किं  बहुना । परस्परं द्वैधम् उत्पन्नम् ।
\end{document}

When I typeset this, even when the output is visually fine, trying to copy the text from the PDF gives incorrect results each time. I've tried with both xelatex and lualatex, with four fonts all generously available online for free: Chandas, Noto Sans Devanagari, Noto Serif Devanagari, Adishila:

• Correct text:

  • किं बहुना । परस्परं द्वैधम् उत्पन्नम् ।

• xelatex:

  • कं बहुना । परɕपरं जैधम् उɊपਯम् । (Chandas)
  • ɫकʌ बहुना । परȺरं द्वैधम् उत्पȡम् । (Noto Sans Devanagari)
  • ȫक बहुना । परस्परं द्वैधम् उत्पन्नम् । (Noto Serif Devanagari)
  • िकं बहुना । परस्परं द्वैधम् उत्पन्नम् । (Adishila)

• lualatex:

  • िकं बहुना । पर�परं द्वैधम् उ�पन्नम् । (Chandas)
  • िकं बहुना । परस्परं द्वैधम् उत्पन्नम् । (Noto Sans Devanagari — also, the output is broken)
  • िकं बहुना । परस्परं द्वैधम् उत्पन्नम् । (Noto Serif Devanagari — also, the output is broken)
  • िकं बzना । परस्परं द्वैधम् उत्पन्नम् । (Adishila)

So none of these are correct, though for some combinations, only the first syllable was problematic. (It doesn't matter that it's the first syllable; किं anywhere has the same issue.)

(Aside: This was using TeX Live 2020 so lualatex uses LuaHBTeX… yet the output is incorrect compared to xelatex for two of the fonts.)

Is there a way of getting the correct text to be copied?

I also tried wrapping every word using the accsupp package, like \BeginAccSupp{ActualText=किं}किं\EndAccSupp{} and so on, but that results in complete gibberish.

Answer 1

When I compile with LuaLaTeX on TeX Live 2020, I get this when I copy and paste:

किं बहुना । परस्परं द्वैधम् उत्पन्नम् ।

From the following MWE, which should also save you some typing if you also want to define \devanagarifontsf, \devanagarifonttt, etc.

\documentclass{article}
\tracinglostchars=2

\usepackage{iftex}
\usepackage{polyglossia}

\ifluahbtex
  \defaultfontfeatures{ Renderer=HarfBuzz, Ligatures=TeX }
\fi

\setmainlanguage{sanskrit}
%% Font available at:
%% https://www.google.com/get/noto/#serif-deva
\newfontfamily\devanagarifont[Script=Devanagari]{Noto Serif Devanagari}

\begin{document}
किं  बहुना । परस्परं द्वैधम् उत्पन्नम् ।
\end{document}

This still does not, however, copy-paste correctly from XeLaTeX.

I normally use babel, which among other things can auto-detect the language I type in, but the solution is very similar.

As Marcel Kruger said in the comments, modern LuaLaTeX supports HarfBuzz rendering, but you need to select it.