1

I have a very simple test case that displays all of the Latin-1 printable characters (U+0021 - U+007e, U+00A1 - U+00ff). It works correctly for every UTF-8 character except U+00AD. It fails with both LuaLaTeX and XeLaTex.

It does not matter if I use unicode-math, nor does it matter if I set the monospace font. If I open the source file in Notepad, everything displays correctly, no matter which font I choose (as long as Latin-1 is supported by the font). If I do set a font using setmonofont it doesn't make any difference.

Here is the test case:

\documentclass[10pt]{article}
\begin{document}
\begin{verbatim}
! " # $ % & ' ( ) * + , - . /
0 1 2 3 4 5 6 7 8 9 : ; < = > ?
@ A B C D E F G H I J K L M N O
P Q R S T U V W X Y Z [ \ ] ^ _
` a b c d e f g h i j k l m n o
p q r s t u v w x y z { | } ~

¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ­ ® ¯
° ± ² ³ ´ µ ¶ · ¸ ¹ º » ¼ ½ ¾ ¿
À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï
Ð Ñ Ò Ó Ô Õ Ö × Ø Ù Ú Û Ü Ý Þ ß
à á â ã ä å æ ç è é ê ë ì í î ï
ð ñ ò ó ô õ ö ÷ ø ù ú û ü ý þ ÿ
\end{verbatim}
\end{document}

This does not show correctly in the question, so an image of the source is also included in a monospace font.

Sample Test Code

And here is the output. It can be seen that on the first line above the ASCII range (starting with the inverted !) that following the "not" symbol, the hyphen is omitted. This may be correct in non-verbatim mode, since it is a "weak" hyphen if it were embedded in a word. However, that processing should not occur in verbatim mode. Note that it also fails when not in verbatim mode, and since the weak hyphen is not embedded in a word, it should not be omitted (as Notepad shows).

Test Output

0

3 Answers 3

6

Whatever Notepad shows should not be taken as an absolute rule. For instance, the editors I have on my machine don't show U+00AD.

enter image description here

Not even if I ask to show invisible characters

enter image description here

However, you might decide to show the character inside verbatim and it's not difficult: hook into the code to add your personal stuff when initiating verbatim.

\documentclass[10pt]{article}
\usepackage{etoolbox}

\makeatletter
\patchcmd{\@verbatim}{\@noligs}{\@noligs\@otherstuff}{}{}
\def\@otherstuff{\catcode"AD=\active}
\begingroup
\catcode"AD=\active
\gdef^^ad{{\ooalign{-\cr\hidewidth?\hidewidth\cr}}}
\endgroup


\begin{document}
\begin{verbatim}
  ! " # $ % & ' ( ) * + , - . /
0 1 2 3 4 5 6 7 8 9 : ; < = > ?
@ A B C D E F G H I J K L M N O
P Q R S T U V W X Y Z [ \ ] ^ _
` a b c d e f g h i j k l m n o
p q r s t u v w x y z { | } ~

  ¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ­ ® ¯
° ± ² ³ ´ µ ¶ · ¸ ¹ º » ¼ ½ ¾ ¿
À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï
Ð Ñ Ò Ó Ô Õ Ö × Ø Ù Ú Û Ü Ý Þ ß
à á â ã ä å æ ç è é ê ë ì í î ï
ð ñ ò ó ô õ ö ÷ ø ù ú û ü ý þ ÿ
\end{verbatim}
\end{document}

enter image description here

Decide about the appearance, here a question mark superimposed to a hyphen.

3

This is expected behavior. Notepad might be showing you the character as a hyphen, but my editor for one does not: It displays <0xad> to denote an unprintable character.

You see, back in 2002 when Unicode 4.0 first came out, the category of U+00AD SOFT HYPHEN was changed from the previous "Pd" (punctuation, dash) to "Cf" (other, format), following HTML 4.0's lead, where &shy; has always behaved that way.

This means that the sole purpose of this character is to mark an invisible hyphenation opportunity, which only becomes a visible hyphen at the end of a line after formatting, if and when needed. Really just like \- in TeX, all things considered.

enter image description here

4
  • I would agree, except that the character is inside a verbatim environment, which is intended to display its contents exactly as they are. Also, Word 2016 displays the character. I believe that, in isolation, or in the verbatim environment, that the character should be displayed. May 15, 2022 at 17:35
  • I disagree, as there is simply no character to display here. How do you “print“ a format character, anyway? By my latest count, Unicode knows 161 “formatting” characters. SOFT HYPHEN is the only one to be included in the LATIN-1 SUPPLEMENT block, but the BASIC MULTILINGUAL PLANE has many more: There's ZERO WIDTH JOINER (U+200D) and RIGHT-TO-LEFT MARK (U+200F) and 159 more – none of which will print because they are, by definition, unprintable. Not exactly a new issue, either. ASCII always had (unprintable) control characters like NUL (U+0000), ALERT (U+0007) etc.
    – Ingmar
    May 15, 2022 at 18:01
  • First, there certainly is a character to be printed. If it used, as intended, in text as soft hyphenation, and the containing word breaks at that point, then it will be displayed. Second, in verbatim mode, printable characters are to be displayed without invoking editing or formatting. Otherwise, you cannot actually see what is there. Third, it is not a control character, of which there are 65 in Latin-1, it is a printable character that may be suppressed when hyphenating a word. Outside of that use, it is still printable. May 15, 2022 at 21:59
  • No, there is no character to be printed here. According to the Unicode specs it's “an invisible format character indicating a possible hyphenation location” (my emphasis). Note how it doesn't say control character, although the difference doesn't really matter much in this context. I have amended my answer to include a screenshot from the Unicode website, showing the details (Note the dashed bounding box used for unprintable characters). That said, I really have zero interest in discussing this any further. What you are experiencing is documented and expected behavior. Deal with it.
    – Ingmar
    May 16, 2022 at 6:44
1

A regular expression can find formatting characters and replace them with something visible:

visible now

MWE

\documentclass{article}
\usepackage{xparse}
\usepackage{xcolor}

\ExplSyntaxOn

\NewDocumentCommand { \showAD } { +m } {
    \tl_set:Nn \l_tmpa_tl { #1 }
    \regex_replace_all:nnN 
                { \xAD } 
                { \cB\{ \c{color} \cB\{ red \cE\} A \cE\} \c{llap}\cB\{ D \cE\} } 
                \l_tmpa_tl
    \tl_use:N \l_tmpa_tl
}
                        
\ExplSyntaxOff

\begin{document}

abc­xyz (invisible)

\showAD{abc­xyz} (visible)

\ttfamily
\showAD{¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ­ ® ¯}

\end{document}
10
  • I'm totally gonna bookmark this for later use … Thanks :)
    – Ingmar
    May 17, 2022 at 13:08
  • Certainly I could have used a regex to find and replace instances of U+00AD with U+002D, but I knew exactly where the character was located. I inserted it by hand, in hex, as I all of the other characters. I did not want to replace it, I wanted it to be displayed in the verbatim environment. I accepted the work-around to "tweak" the verbatim environment (slightly modified), but I still think that the verbatim environment should not perform any processing on the characters. Simply due to its entire reason for existing -- that of displaying source code. May 18, 2022 at 21:30
  • @MichaelLeeFinney May be X-Y. If the chracter is for control, not display, how will any environment display it without tweaking/processing? This character, between the brackets, >‍<, displays as invisible on my setup. So verbatim doing "no processing" and displaying nothing would be the desired behaviour, correct? Now, marking the character (by a space, say; or by nothing in a grid; or by < and >) is different, but that counts as a type of processing, doesn't it?
    – Cicada
    May 19, 2022 at 1:41
  • @MichaelLeeFinney You may be thinking verbatim is like a hex dump, perhaps. Verbatim doesn't tokenize, true, but it also just follows the font. (was designed especially to keep whitespace: space, tab, EOL, etc - all of which turn into glue when tokenized).
    – Cicada
    May 19, 2022 at 1:44
  • For controls, spaces, etc. you are correct. In the case of U+AD, it is a soft hyphen and may or may not display depending on if it is chosen as a hyphenation point. When in isolation, and especially when in verbatim mode, then it should display. If you are looking at source text, how will you know that you have a soft hyphen in the text unless you can see it being displayed? The only other choice is literally a hex dump, which is not what most people want to see. There must be some mode where the soft hyphen always displays, and for LaTeX, I believe that mode is the verbatim environment. May 20, 2022 at 4:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .