2

This is not a "how-to" question but rathe a "why" question.

I'm working on re-writing a MS Word document in LaTeX to be compiled with XeTex. I noticed some symbols not showing, especially a µ (or "micro") symbol, sometimes showing and sometimes not. Following mcve shows the behaviour (Note: Compiling with Pdflatex will throw an ! Package inputenc Error: Invalid UTF-8 byte sequence. error on several lines.)

\documentclass[11pt,a4paper]{scrarticle}

\begin{document}
\ \\
GREEK SMALL LETTER MU: μ\\
MICRO SIGN: µ\\
German Umlaut: äöü ÄÖÜ ß\\
Some greek letters: αΑ βΒ \\
mathematical symbols: 𝛂, 𝛼 \\ 
\end{document}

The Output looks like this:

enter image description here

Clearly the first instance of µ is missing. After some digging, I found that those two symbols translate to different Ansi in Notepad++, the first being μ, the second being µ.

Why does the first µ fail to render but not the second? Why do greek capital letters render but not small ones? I'm pretty sure it has to do with encodings but I fail to understand exactly what's going on.

Edit: The answer of Chris H below seems to confirm different unicode characters being the culprit, so I updated the example with more unicode characters that reproduce the behaviour. The reason why some unicode characters render and others don't is still not clear to me.

3
  • 4
    By chance I was looking in something completely different but also involving the Micro Sign / Greek mu. The Micro Sign is in unicode terms u+00b5 whilst the greek mu is u+03bc (see e.g. fileformat.info/info/unicode/char/00b5/index.htm and fileformat.info/info/unicode/char/03bc/index.htm) In your comments these 2 look identical but (verified by an octal dump and shown by you in notepad++ as well) are different, so probably here lies your problem.
    – albert
    Mar 1, 2021 at 12:14
  • Obviously that IS the source of my problem. I'm just wondering why some unicode symbols render and others don't. Is that due to a missing representation of that unicode symbol in the default font? I edited my code example to show some more unicode symbols that don't render and others that do.
    – Dschoni
    Mar 1, 2021 at 15:25
  • Actually it not only happens in the lstlistings package, but also in regular text - without any warning. Editing my example to be "real minimal".
    – Dschoni
    Mar 1, 2021 at 15:32

2 Answers 2

4

A couple of tools combine to figure out what's going on. First string-functions.com, converts your Ansi strings to the seemingly identical μ and µ respectively (set the input encoding to ISO8859-1 and the output to UTF-8). Copy-pasting from your quoted source also works, but I wanted to start from the known-different characters. Pasting those two mus into a Python3 script gives:

>>> print (hex(ord("μ")))
0x3bc
>>> print (hex(ord("µ")))
0xb5

Confirming, as in Albert's comment, that the first is GREEK SMALL LETTER MU and the second is MICRO SIGN (in caps according to the Unicode standards).

So listings is rendering micro signs successfully, but not Greek letters. Unicode allows fonts to use the same glyph for multiple codepoints, so they can indeed be pixel-identical.


Here's a Python script to check for Unicode characters in the whole source (pasted into the multi-line string source, note the r''' to start a raw multiline string and ''' to end a multiline string.

import unicodedata
source=r'''\documentclass[11pt,a4paper]{scrarticle}

\usepackage{xcolor}
\usepackage{listings}

\lstset{
    frame=trBL, 
    language=C++, 
    basicstyle=\small, % make fontsize a little smaller to better fit code
    showstringspaces=false, % don't mark spaces in strings
    %numbers=left, % display line numbers on the left
    commentstyle=\color{teal}, % comment color
    keywordstyle=\color{blue}, % keyword color
    stringstyle=\color{red} % string color
}

\begin{document}
\begin{lstlisting}
public Byte SomeCurrent;  //Current measured in μA
public Byte OtherCurrent; //Other current in µA
\end{lstlisting}
\end{document}'''

header=["char","codepoint","position", "name"]
print (header)
for i,char in enumerate(source):
    if ord(char)>127:
        line=[char, hex(ord(char)), i, unicodedata.name(char)]
        print (line)

The output it gives is:

['char', 'codepoint', 'position', 'name']
['μ', '0x3bc', 542, 'GREEK SMALL LETTER MU']
['µ', '0xb5', 590, 'MICRO SIGN']

It would be easy to read in the source from a file, and to give the line number and position within the line of each unicode character, but I'll leave that as an exercise for the reader.

4
  • This isn't surprising when the default way of entering non-keyboard characters in Word (and indeed on Windows in general) is visual. To an earlier writer it looked right. I'm on Linux and type some Unicode characters (like µ, °, and ×) from memory; for Greek I use a cheat sheet, but either way it's keyboard->codepoint
    – Chris H
    Mar 1, 2021 at 13:02
  • I'm still puzzled, why the micro sign renders correctly and the greek small letter doesn't. I naively thought, Xetex would get right of that kind of encoding mess automagically... Fortunately, the µ I have on my keyboard is the one that renders ;)
    – Dschoni
    Mar 1, 2021 at 15:13
  • @Dschoni having micro-mu is a big advantage of German keyboard layouts. I don't think Xetex and listings have much choice, as the glyph might be separate, but I suspect the issue is mainly listings. It's commonly used with pdflatex and didn't originally have (much) support for unicode
    – Chris H
    Mar 1, 2021 at 15:36
  • I tested in regular text and it is not limited to lstlistings. Updated my question.
    – Dschoni
    Mar 1, 2021 at 15:41
1

After a little bit of more investigation, I could reproduce the behaviour outside of lstlisting as well. This brought me to this question. The answer is, that the font I'm using is missing these unicode characters, which is also shown in the extended log - but not shown as a warning when compiling, such as:

Missing character: There is no μ in font [lmroman10-regular]:mapping=tex-text;!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .