# How do I encode foreign characters so that they are searchable in the resultant PDF file?

This may be a PDF problem rather than a LaTeX problem, but as it involves pdfLaTeX output I'm hoping I won't get my wrist slapped too hard.

I'm creating a bibliography, and inevitably some of the authors' names have accented or other foreign characters in them. I'd like to be able to search the PDF file for these authors, but I'm having problems with some of the characters. As an example, I've created a simple .tex file as follows:

\documentclass{article}
\usepackage[utf8]{inputenc}
\begin{document}
Some foreign characters: öøäéüæåñ
\end{document}


I can search successfully in the resultant PDF file for ö, ä, é, ü, å and ñ, either using those specific characters or an unaccented o, a, e, u or n. However, the PDF file will not find the ø or æ characters whether I copy and paste them into the search box from another programme, or whether I use Alt+155 or Alt+145. The problem is compounded by the fact that some journals use a simple 'o' for ø or 'ae' for æ, so I'd like to get a search 'hit' for ø using either ø or o, and similarly a search hit for æ using either æ or ae. Is any of this remotely possible, or am I trying to be too clever?

-
So you do get the behavior you want with these characters: ö, ä, é, ü, å and ñ, but not with ø and æ, is that correct? – Tim A Jan 16 '12 at 20:38
Absolutely correct @TimA, that's a much more concise summary of my problem! – Mark Birtwistle Jan 16 '12 at 21:33
This really looks like a duplicate of one of the above linked questions. Please have a look at it and tell us if they are sufficient for you. We then can close this question as a duplicate. Thanks. – Martin Scharrer Jan 17 '12 at 11:01
@MartinScharrer, these questions are helpful, but do not 100% solve my problem. I still cannot get a resultant PDF file to search for Grønning with Gronning – something that EndNote for example will do. Is this better asked as a separate question? – Mark Birtwistle Jan 21 '12 at 11:26

Can you try:

\usepackage[T1]{fontenc}


Full test document:

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\begin{document}
Some foreign characters: öøäéüæåñ
\end{document}

-
Thanks @knut, that goes pretty close. I'm annoyed I didn't think of it myself. The PDF will now find the ø and æ characters, and will even find æ with 'ae'. However, it will not find ø with 'o'. As an example, here is a small document with real authors' names: – Mark Birtwistle Jan 16 '12 at 22:26
\swearword'. This is the second time I've hit 'Enter' by mistake and then been timed out when I've tried to edit the comment! Anyway @knut, by way of an example, I can find 'Fjære' by searching with 'Fjære' or 'Fjaere' and I can find 'Grønning' by searching with 'Grønning'. However, I can't find 'Grønning' by searching with 'Gronning'. Maybe the last is just a step too far...? – Mark Birtwistle Jan 16 '12 at 22:37

Although I can't give a working solution, I want to explain where the problem stems from.

Like Crissov conjectured in his comment, this has a lot to do with the reader application you are using. In my case Okular would not even find öaéuåñ inside of words. As example I used:

\documentclass{article}
\usepackage{fontspec}
\setmainfont[Ligatures=TeX]{STIXGeneral}
\XeTeXinputnormalization=1 %or 2
\begin{document}
Töst Tøst Täst Tést Tüst Tæst Tåst Testeñ
\end{document}


With \XeTeXinputnormalization=1 I can't find the special characters at all, when searching for regular ones. If I set \XeTeXinputnormalization=2, at least I'll find To in Töst. Still I will find neither Tost nor To in Tøst.

The reason for this behaviour is, that Okular is doing a quite exact comparison and, more importantly, there is a major difference between the characters ö and ø (which causes your problem):

ö (U+00F6) has a decomposition into a base character o (U+006F) and an accent ̈ (U+0308) associated to it. While you can write o̷ (U+006F U+0337) and it will usually displayed as ø, it is not associated to ø (U+00F8) and when converting between unicode normal forms it will not be converted. For æ it is the same as for ø. All the other characters you mentioned do have an associated decomposition. (See unicode character decomposition mapping for reference.)

So it seems your reader application does the sensible thing, to consider not only the actual character, but also the base character of its decomposition. For the other ø, æ etc. it is simply too much to expect from it.

You could probably convert all ø to images and only lay an invisible o behind it, like it is done for scanned pdfs with OCR. But, apart from that being an ugly solution, you would not be able to find the word, if you do type the actual ø then...

-
If I use the most recent version of the STIX fonts (with \setmainfont[Ligatures=TeX]{STIX}, my previewer is able to find all accents, with any value of \XeTeXinputnormalization`. If Okular isn't able to find them, then it's a problem with that previewer. – egreg Feb 8 '14 at 14:28
@egreg: Yes it is a missing feature in Okular (bugs.kde.org/show_bug.cgi?id=274933) for the accents. – canaaerus Feb 8 '14 at 14:42