8

I have several issues with searching and/or copy-pasting text from my PDF created with XeLaTex. Consider the following MWE, the PDF search gives me no results for certain words.

\documentclass[%
   headsepline,
   footsepline,
   twoside,       
   numbers=noenddot,
   fontsize=11pt]{scrbook}
\usepackage[a4paper]{geometry}%
\usepackage{fontspec}
\setmainfont{Cambria}

\makeatletter\chardef\l@nohyphenation=255 \makeatother
\usepackage[ngerman=nohyphenation]{hyphsubst}
\usepackage[ngerman]{babel}
\usepackage{microtype}

\newcommand{\name}[1]{\textsc{#1}}

\begin{document}
\sloppy

\name{Albert Einstein} veröffentlichte 1905 die spezielle 
und 1916 die allgemeine Relativitätstheorie. 

Aus dem Relativitätsprinzip folgt unmittelbar, dass es keine Möglichkeit gibt, 
eine absolute Geschwindigkeit eines Beobachters im Raum zu ermitteln 
und damit ein absolut ruhendes Bezugssystem zu definieren. 

Übersetzt man die Beobachtungen des ruhenden Beobachters über eine
\name{Lorentz}-Transformation in die des Bewegten, so stellt sich heraus, 
dass dieser neben der elektrischen Kraft eine weitere, magnetische, Kraft wahrnimmt.

\end{document}
  1. Words with small caps are not found, I guess, because there is no definition of a font "Cambria-smallcaps" or something like that. I get searchable and copyable small caps in the PDF, when I use another font which has small caps defined, for example, something like \setmainfont[SmallCapsFont={Latin Modern Roman Caps}]{Cambria}. However, I don't feel comfortable using several fonts in one document. So another method to achieve searchable and copyable small caps is to add this:

    \usepackage{accsupp}
    \newcommand*{\copyablesc}[1]{%
        \BeginAccSupp{%
            ActualText=\detokenize{#1},%
            method=escape,
        }%
        \textsc{\lowercase{#1}}%
        \EndAccSupp{}%
    }
    

    With \newcommand{\name}[1]{\copyablesc{#1}} I now get the names in lowercase small caps. This means that the uppercase smallcaps i.e. capitalization (e.g. at beginning of the words/names) are ignored which is not what I want. Is there a way to define \copyablesc in a way it functions/looks exactly like the usual \textsc or another way to redefine \textsc so that it is found in PDF search without changing the font?

    EDIT: I just recognized that I get the desired result by leaving \lowercase and just typing \textsc{#1}% in the above code. I am sorry for the mistake.

  2. Words with ligatures (e.g. the "fi" in "definieren") are not found in the PDF and when I copy-paste them, I get a dummy symbol instead. I tried cmapand mmap package and \input{glyphtounicode}, but these don't work with XeLaTex. So the only way to make ligatures searchable/copyable obvioulsy is to switch them off completely with \setmainfont[Ligatures={NoCommon, Tex}]{Cambria}. But I would really like to keep them because it looks nicer. Is there another way to keep ligatures and make them searchable/copyable in PDF?

  3. In contrary to ligatures, umlaute (ä, ö, ü) are somewhat treated like the behavior I would want for ligatures. This means, they behave like a melt of two signs.

    • When I copy-paste them from PDF into MS Word the uncapitalized umlaute (ä, ö, ü) are treated like letters with a math accent, meaning that in MS Word all words containing them are marked as spelled incorrectly and that when deleting them I have to delete the accent first and the letter second. When I search for umlaute (e.g. for "ö") in PDF, every "o" is found, which is not correct. These umlaute are single signs in German and should be treated as such. Searching for "ö" should find only "ö" and not "o".

    • Capitalized umlaute (Ä, Ö, Ü) are not searchable/copyable at all and copy-pasting results in "A, O, U", respectively, followed by a dummy sign.

    So is there any way to tell XeLaTex to treat umlaute as one sign/ligature and turn them searchable/copyable as such?

  • 4
    Not really. If you try with another font, e.g. the default Latin Modern, most of your points will work. So it is a problem with Cambria and xetex doesn't has hooks to patch a buggy font. With lualatex one could perhaps do something. Regarding the search for ö: this is due to the pdf-viewer. If it decides to search also for o or O, neither xetex nor luatex will be able to prevent this. – Ulrike Fischer Sep 15 '17 at 13:20
  • Ok, that is what I was apprehensive of. It means that I would have to change the font. I have written more than 100 pages of my thesis now and have a lot of graphical material. Changing the font would mean that I would have to rework all my pictures. Do you know any font which has the features I want and looks similar to Cambria? Regarding LuaLaTex... I tried to compile with it, but got the same problems. So what would it be that I could do differently with LuaLaTex? I never used it before. – Lysanne Sep 15 '17 at 13:35
  • 3
    Charis SIL (a charter clone) is imho similar. But I didn't test if is has similar problems. And with lualatex you can patch fonts (see e.g. tex.stackexchange.com/questions/385089/big-lvert-is-too-big/…), but I didn't try yet if one can correct the cambria problems. – Ulrike Fischer Sep 15 '17 at 14:46
  • Thank you. I tried Charis SIL. I also did some research on Cambria clones and found Caladea. This font almost looks exactly like Cambria. Caladea and Charis SIL do work regarding the issues I addressed above, but however, both fonts lack small caps which are typed as normal text. – Lysanne Sep 16 '17 at 11:48
  • 1
    I just found out that adding \XeTeXgenerateactualtext=1 makes the small caps and ligatures searchable without the use of switching them off. As outlined here the selection boxes are located incorrectly. Nevertheless copy-pasting still works. – Lysanne Sep 16 '17 at 14:22
4

As I wrote in the comment: with lualatex it is probably possible to patch the font. As a proof of concept:

\pdfvariable compresslevel 0
\documentclass[%
 fontsize=11pt,%
  ngerman,
]{scrbook}

\usepackage{luacode}

\begin{luacode}

local patch_cambriasc = function (fontdata)
 if fontdata.fontname == "Cambria"
 then
   fontdata.descriptions[983054]["unicode"]=109
   fontdata.descriptions[983055]["unicode"]=110
   fontdata.descriptions[983056]["unicode"]=111
   fontdata.descriptions[983213]["unicode"]=776 -- accent 0308
   fontdata.descriptions[983219]["unicode"]=778 -- accent 030A
   fontdata.descriptions[983078]["unicode"]=230 -- æ  
   fontdata.descriptions[983084]["unicode"]=231 -- ç
 end
end    

luatexbase.add_to_callback
 (
  "luaotfload.patch_font",
  patch_cambriasc,
  "change_cambria"
 )
\end{luacode}

\usepackage{fontspec}

\setmainfont[Ligatures={NoCommon}]{Cambria}
\newcommand{\name}[1]{\textsc{#1}}

\begin{document}
mno \name{omn}
\end{document} 

When compiled with luatex I get in the pdf tounicode entries for all glyphs:

7 beginbfchar
<008F> <006D>
<0090> <006E>
<0091> <006F>
<0112> <006D>
<0113> <006E>
<0114> <006F>
<0373> <0031>
endbfchar

and the text copies and paste fine.

The following variant of the lua patch works too:

local patch_cambriasc = function (fontdata)
 if fontdata.fontname == "Cambria"
 then
   fontdata.characters[983054]["tounicode"]="006D"
   fontdata.characters[983055]["tounicode"]="006E"
   fontdata.characters[983056]["tounicode"]="006F"
   fontdata.characters[983213]["tounicode"]="0308" -- accent 0308
   fontdata.characters[983219]["tounicode"]="030A" -- accent 030A
   fontdata.characters[983078]["tounicode"]="00E6" -- æ
   fontdata.characters[983084]["tounicode"]="00E7" -- ç
 end
end

But I have no idea with the unicode fields works only in the description table and the tounicode only in characters.

  • Yes, your example works. I tested again and typed Mustermann \name{Mustermann} in the document. This gives Mustermann M􀄚􀄘􀄙􀄊􀄗m􀄆nn when I copy and paste it. I never worked with LuaLaTeX and don't know how the luacode works. Do I have to patch something more in there? – Lysanne Mar 16 '18 at 19:16
  • Well you need more of fontdata.descriptions lines -- for every char you need one. – Ulrike Fischer Mar 16 '18 at 19:18
  • Ok, I see, the number at the end of the line is the unicode character. Where do I get the number in the first brackets of the fontdata.description from? Can you please point me somewhere, where I can read how this works? – Lysanne Mar 16 '18 at 19:47
  • 1
    I made a fontdumb (see the luaotfload manual), I made an uncompress pdf, in the pdf I looked up the index number of the char m (0112), converted this from hex to decimal (274) and then searched in the fontdumb for ["index"]=274, – Ulrike Fischer Mar 16 '18 at 19:52
  • 2
    I added four more values. The diaresis and the ring are accents in cambria, not glyphs. – Ulrike Fischer Mar 18 '18 at 15:55
5

Ulrike already provided a good answer. Let me provide some useful information.

If you are using Open Type fonts that are designed for Unicode, writing your document in Unicode, and using a Unicode-aware TeX engine with fontspec:

In a well-designed Unicode Open Type font, the character name of a ligature is of the form hello_there. The underscore connects a character named hello with one named there. More than two may be involved. So, the ligature between f and i is named f_i. A few ligatures (such as fi) have their own names, dating back decades.

Substitution of characters with ligatures is specified within the font, and understood by fontspec. However, that information is already processed by the time you have the PDF.

Ligatures can be assigned to the private/corporate user area in Unicode, and some fonts do that. But the standard prefers that ligatures do not have a code point. Instead they are found by reference.

When a good PDF reader sees the character f_i, it knows that it is supposed to see the two characters f and i when searching, and it knows that it is supposed to provide those two characters in plain text output. The reasons are that not all fonts have f_i, and those that have it may use different code points.

Unfortunately, some PDF readers (and text extractors) do not see f_i as two characters. They see it as a single character, which cannot be found by search, and cannot be exported as two characters to plain text.

Something similar applies to small caps. The word WHAT in small caps should be searched as ordinary what, and exported as ordinary what. Good PDF readers do that. But others see only the non-standard code points of the small cap letters.

The bottom line: This is not a property of XeTeX or LuaLaTeX. It is a property of the software that views the PDF.

Old-fashioned non-Unicode pdflatex, with fontenc and all that, is a different issue.

EDIT: A nasty font designer could use serdkwul as the name of the ligature between f and i. That is technically allowed, and will work as long as the ligature is defined within the font. It will also display and print properly in PDF. However, since the character name is not f_i, PDF readers will not know that it is supposed to be decomposed to f and i.

EDIT2: As for small caps, or any other variation on a character: In a well-designed Open Type font, variations of a character have the same name as the base character, followed by an extension. So, the small cap version for a might be named a.sc or a.smcp or something similar. The font's own lookup tables will state which character to use, when small caps are requested. A good PDF reader knows that a.ext is a variation on a. It will find it as a in search, and export it as a in plain text. A badly designed font will use something obscure such as asmcp (without the dot) for the lookup. That will display correctly in the PDF, but not be searched or exported as a. An inadequate PDF reader will not understand that a.sc is a variation of a. Character name a.foo.bar is also legal, as a variation on a.

Although a small cap letter is likely to have extension .sc or .smcp in the font, that is not a requirement. So, it is not reliable for a PDF reader to find small caps letters merely by looking at the extension. It may be that a high-end PDF reader can do it, if the original font is installed, so that the reader can internally inspect the font. I'm not sure.

  • Thank you for the background information! The font I use (Cambria) doen't provide small caps. fontspec creates them somehow I think. I would wonder if it named them illogically. I also recognized that although Cambria has e.g. the fi ligature (I can find it in the symbols table in MS Word) it is presented as normal f and i in the PDF. When I copy it from PDF it is taken for a ligature or unknown character. So obviously XeLaTeX seems to put the ligatures in, but they're not represented as such by the PDF reader because e.g. the name might be strange, following your explanation. – Lysanne Mar 19 '18 at 17:52
  • @Lysanne Actually, the Cambria font (Microsoft) does have small caps. Just looked inside it, with a font editor. As you noted, it has fi but not f_i. If you copy and paste to plain text (Unicode aware), and if the font in your text editor is Cambria, then you should see fi. Copy/paste transfers code points, rather than characters. If the PDF had f_i (and it is a good PDF reader) it will decompose to f and i when you copy. But good or bad, it may copy fi as the single character, which will only appear correct when the text has that character at that exact code point. – user139954 Mar 19 '18 at 18:07
  • Oh ok, I looked in my fonts library and couldn't find the small caps file, so I thought it is not provided by MS. But yes, you're right, I just checked. When I insert the ligature in my text editor, compile and copy from PDF, I get plain f and i. That means that my PDF reader is really lacking f_i. I think I understand the problem now. However, most of my colleagues use the same PDF reader, so I have to adjust to that. – Lysanne Mar 19 '18 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.