Searchable and copyable small caps, ligatures and umlaut in PDF created by XeLaTex

Question

I have several issues with searching and/or copy-pasting text from my PDF created with XeLaTex. Consider the following MWE, the PDF search gives me no results for certain words.

\documentclass[%
   headsepline,
   footsepline,
   twoside,       
   numbers=noenddot,
   fontsize=11pt]{scrbook}
\usepackage[a4paper]{geometry}%
\usepackage{fontspec}
\setmainfont{Cambria}

\makeatletter\chardef\l@nohyphenation=255 \makeatother
\usepackage[ngerman=nohyphenation]{hyphsubst}
\usepackage[ngerman]{babel}
\usepackage{microtype}

\newcommand{\name}[1]{\textsc{#1}}

\begin{document}
\sloppy

\name{Albert Einstein} veröffentlichte 1905 die spezielle 
und 1916 die allgemeine Relativitätstheorie. 

Aus dem Relativitätsprinzip folgt unmittelbar, dass es keine Möglichkeit gibt, 
eine absolute Geschwindigkeit eines Beobachters im Raum zu ermitteln 
und damit ein absolut ruhendes Bezugssystem zu definieren. 

Übersetzt man die Beobachtungen des ruhenden Beobachters über eine
\name{Lorentz}-Transformation in die des Bewegten, so stellt sich heraus, 
dass dieser neben der elektrischen Kraft eine weitere, magnetische, Kraft wahrnimmt.

\end{document}

1. Words with small caps are not found, I guess, because there is no definition of a font "Cambria-smallcaps" or something like that. I get searchable and copyable small caps in the PDF, when I use another font which has small caps defined, for example, something like \setmainfont[SmallCapsFont={Latin Modern Roman Caps}]{Cambria}. However, I don't feel comfortable using several fonts in one document. So another method to achieve searchable and copyable small caps is to add this:

   \usepackage{accsupp}
   \newcommand*{\copyablesc}[1]{%
       \BeginAccSupp{%
           ActualText=\detokenize{#1},%
           method=escape,
       }%
       \textsc{\lowercase{#1}}%
       \EndAccSupp{}%
   }
   

   With \newcommand{\name}[1]{\copyablesc{#1}} I now get the names in lowercase small caps. This means that the uppercase smallcaps i.e. capitalization (e.g. at beginning of the words/names) are ignored which is not what I want. Is there a way to define \copyablesc in a way it functions/looks exactly like the usual \textsc or another way to redefine \textsc so that it is found in PDF search without changing the font?

   EDIT: I just recognized that I get the desired result by leaving \lowercase and just typing \textsc{#1}% in the above code. I am sorry for the mistake.

2. Words with ligatures (e.g. the "fi" in "definieren") are not found in the PDF and when I copy-paste them, I get a dummy symbol instead. I tried cmapand mmap package and \input{glyphtounicode}, but these don't work with XeLaTex. So the only way to make ligatures searchable/copyable obvioulsy is to switch them off completely with \setmainfont[Ligatures={NoCommon, Tex}]{Cambria}. But I would really like to keep them because it looks nicer. Is there another way to keep ligatures and make them searchable/copyable in PDF?

3. In contrary to ligatures, umlaute (ä, ö, ü) are somewhat treated like the behavior I would want for ligatures. This means, they behave like a melt of two signs.

   • When I copy-paste them from PDF into MS Word the uncapitalized umlaute (ä, ö, ü) are treated like letters with a math accent, meaning that in MS Word all words containing them are marked as spelled incorrectly and that when deleting them I have to delete the accent first and the letter second. When I search for umlaute (e.g. for "ö") in PDF, every "o" is found, which is not correct. These umlaute are single signs in German and should be treated as such. Searching for "ö" should find only "ö" and not "o".

   • Capitalized umlaute (Ä, Ö, Ü) are not searchable/copyable at all and copy-pasting results in "A, O, U", respectively, followed by a dummy sign.

   So is there any way to tell XeLaTex to treat umlaute as one sign/ligature and turn them searchable/copyable as such?

Answer 1

As I wrote in the comment: with lualatex it is probably possible to patch the font. As a proof of concept:

\pdfvariable compresslevel 0
\documentclass[%
 fontsize=11pt,%
  ngerman,
]{scrbook}

\usepackage{luacode}

\begin{luacode}

local patch_cambriasc = function (fontdata)
 if fontdata.fontname == "Cambria"
 then
   fontdata.descriptions[983054]["unicode"]=109
   fontdata.descriptions[983055]["unicode"]=110
   fontdata.descriptions[983056]["unicode"]=111
   fontdata.descriptions[983213]["unicode"]=776 -- accent 0308
   fontdata.descriptions[983219]["unicode"]=778 -- accent 030A
   fontdata.descriptions[983078]["unicode"]=230 -- æ  
   fontdata.descriptions[983084]["unicode"]=231 -- ç
 end
end    

luatexbase.add_to_callback
 (
  "luaotfload.patch_font",
  patch_cambriasc,
  "change_cambria"
 )
\end{luacode}

\usepackage{fontspec}

\setmainfont[Ligatures={NoCommon}]{Cambria}
\newcommand{\name}[1]{\textsc{#1}}

\begin{document}
mno \name{omn}
\end{document} 

When compiled with luatex I get in the pdf tounicode entries for all glyphs:

7 beginbfchar
<008F> <006D>
<0090> <006E>
<0091> <006F>
<0112> <006D>
<0113> <006E>
<0114> <006F>
<0373> <0031>
endbfchar

and the text copies and paste fine.

The following variant of the lua patch works too:

local patch_cambriasc = function (fontdata)
 if fontdata.fontname == "Cambria"
 then
   fontdata.characters[983054]["tounicode"]="006D"
   fontdata.characters[983055]["tounicode"]="006E"
   fontdata.characters[983056]["tounicode"]="006F"
   fontdata.characters[983213]["tounicode"]="0308" -- accent 0308
   fontdata.characters[983219]["tounicode"]="030A" -- accent 030A
   fontdata.characters[983078]["tounicode"]="00E6" -- æ
   fontdata.characters[983084]["tounicode"]="00E7" -- ç
 end
end

But I have no idea with the unicode fields works only in the description table and the tounicode only in characters.

Answer 2

Ulrike already provided a good answer. Let me provide some useful information.

If you are using Open Type fonts that are designed for Unicode, writing your document in Unicode, and using a Unicode-aware TeX engine with fontspec:

In a well-designed Unicode Open Type font, the character name of a ligature is of the form hello_there. The underscore connects a character named hello with one named there. More than two may be involved. So, the ligature between f and i is named f_i. A few ligatures (such as fi) have their own names, dating back decades.

Substitution of characters with ligatures is specified within the font, and understood by fontspec. However, that information is already processed by the time you have the PDF.

Ligatures can be assigned to the private/corporate user area in Unicode, and some fonts do that. But the standard prefers that ligatures do not have a code point. Instead they are found by reference.

When a good PDF reader sees the character f_i, it knows that it is supposed to see the two characters f and i when searching, and it knows that it is supposed to provide those two characters in plain text output. The reasons are that not all fonts have f_i, and those that have it may use different code points.

Unfortunately, some PDF readers (and text extractors) do not see f_i as two characters. They see it as a single character, which cannot be found by search, and cannot be exported as two characters to plain text.

Something similar applies to small caps. The word WHAT in small caps should be searched as ordinary what, and exported as ordinary what. Good PDF readers do that. But others see only the non-standard code points of the small cap letters.

The bottom line: This is not a property of XeTeX or LuaLaTeX. It is a property of the software that views the PDF.

Old-fashioned non-Unicode pdflatex, with fontenc and all that, is a different issue.

EDIT: A nasty font designer could use serdkwul as the name of the ligature between f and i. That is technically allowed, and will work as long as the ligature is defined within the font. It will also display and print properly in PDF. However, since the character name is not f_i, PDF readers will not know that it is supposed to be decomposed to f and i.

EDIT2: As for small caps, or any other variation on a character: In a well-designed Open Type font, variations of a character have the same name as the base character, followed by an extension. So, the small cap version for a might be named a.sc or a.smcp or something similar. The font's own lookup tables will state which character to use, when small caps are requested. A good PDF reader knows that a.ext is a variation on a. It will find it as a in search, and export it as a in plain text. A badly designed font will use something obscure such as asmcp (without the dot) for the lookup. That will display correctly in the PDF, but not be searched or exported as a. An inadequate PDF reader will not understand that a.sc is a variation of a. Character name a.foo.bar is also legal, as a variation on a.

Although a small cap letter is likely to have extension .sc or .smcp in the font, that is not a requirement. So, it is not reliable for a PDF reader to find small caps letters merely by looking at the extension. It may be that a high-end PDF reader can do it, if the original font is installed, so that the reader can internally inspect the font. I'm not sure.