# Localized glyphs (locl) have unicode value U+FFFD (�)

Originally this seemed to me like a font issue, as there was no equivalent problem with other fonts, as indicated by this good MWE:

## Copied text from good PDF

EBGaramond12-Italic.otf:
[ - ] бгдпт
[SRB] бгдпт


## Good MWE:

\documentclass[11pt]{article}
\usepackage{fontspec}
\setmainfont{Latin Modern Mono}

\newcommand*{\chars}{бгдпт}

\newcommand*{\samplepars}[1]{{%
\vspace{\normalbaselineskip}%
#1:\par
\fontspec{#1}[Script=Cyrillic]%
{\normalfont [ - ]} {\chars}\par
}}

\begin{document}

\samplepars{EBGaramond12-Italic.otf}%

\end{document}


The problem starts happening with the Source Han fonts (the (""/"K"/"SC"/"TC") fonts support all language glyphs via locl). For the MWE we will use Source Han Sans TC (16MB).

(All of the currently available fonts can be downloaded quickly and easily with these commands.)

If another language than the font file default (e.g. using a language besides Chinese Traditional with the ... TC fonts) is used, substituted glyphs have unicode value U+FFFD � REPLACEMENT CHARACTER. This makes accessibility tools, searching and copying in a PDF file impossible for those glyphs.

The expected behavior would be that the substituted glyphs have the same unicode value as the default glyph they are replacing.

Edit: For comparison, I was able to produce a good CJK PDF with LibreOffice Writer, from this ODT file. Uncompressing it with mutool clean -d -a libreoffice.pdf libreoffice-uncompressed.pdf and viewing the source does not contain any BDC, EMC, or even zh, but one /Lang (en-US). The fonts embedded in the file are converted from OTF to Type1 by LibreOffice, which could be the cause of the glyph unicode values.

If the issue is indeed with TeX, how could it be fixed or worked around?

## Copied text from bad PDF

SourceHanSansTC-Regular.otf:
[ZHS] ���⻣�����贈��⻆���函純賭難海�練�������
[ZHT] 刃令毒骨縣誤船述煙贈雪及角條低詩函純賭難海喝練灰起英次能窗化


\documentclass[11pt]{article}
\usepackage{fontspec}
\setmainfont{Latin Modern Mono}
\setlength{\parindent}{0mm}

\newcommand*{\chars}{%
刃令毒骨縣誤船述煙贈雪及角條低詩函純賭難海喝練灰起英次能窗化}

\newcommand*{\samplepars}[1]{{%
\vspace{\normalbaselineskip}%
#1:\par
\fontspec{#1}[Script=CJK]%
%%{\normalfont [ZHH]} {\addfontfeature{Language=Chinese Hong Kong  }\chars}\par
}}

\begin{document}

\newpage
% \samplepars{SourceHanSerif-Regular.otf}%
% \samplepars{SourceHanSerifK-Regular.otf}%
%%\samplepars{SourceHanSerifHC-Regular.otf}%
% \samplepars{SourceHanSerifSC-Regular.otf}%
% \samplepars{SourceHanSerifTC-Regular.otf}%

\newpage
% \samplepars{SourceHanSans-Regular.otf}%
% \samplepars{SourceHanSansK-Regular.otf}%
%%\samplepars{SourceHanSansHC-Regular.otf}%
% \samplepars{SourceHanSansSC-Regular.otf}%
\samplepars{SourceHanSansTC-Regular.otf}%

\end{document}

• I took a look in the locl-test.pdf provided by Ken Lunde. It has in the page stream BDC/EMC operators which mark up the language and Ken's comment sounds as if this is needed for copy&paste to work correctly. Adding this marks should be possible e.g. with the tagpdf package, but it works currently only for lualatex and pdflatex. But I can't load your font with lualatex, so I can't run more tests. – Ulrike Fischer Sep 19 '18 at 8:36
• I can now run the fonts with luatex (they need a 64bit luatex), but the reference file from Ken is a bit too complicated for real investigations. It would be better to have working example involving only one font and two variants. – Ulrike Fischer Sep 19 '18 at 16:57
• well it is not tagging. But tounicode entries are missing. I asked on the context list for a solution for luatex. – Ulrike Fischer Sep 26 '18 at 21:17
• the context fontloader will be updated to get this working, and I can then import it in lualatex/luaotfload. But I have no idea which of the various libraries in xetex is relevant here. You should try to ask on the xetex mailing list. The discussion on the context list is here mail-archive.com/ntg-context@ntg.nl/msg89086.html – Ulrike Fischer Sep 27 '18 at 9:19
• I have uploaded a new version of luaotfload to ctan which I hope corrects the problem for lualatex. – Ulrike Fischer Oct 3 '18 at 13:42