# How to prevent extendedchars=\true from producing a blank line?

My previous question How to suppress BOM effect in the output? has been solved by @Vaulty. Enabling extendedchars=\true, however, produces an unnecessary blank line at the first line as shown below.

\documentclass{article}
\usepackage{xcolor}
\usepackage{listings}

\lstset
{
breaklines=true,
tabsize=3,
showstringspaces=false,
extendedchars=\true,%<======= SOURCE OF PROBLEM
language={[Sharp]C},
frame=single,
rulecolor=\color{red}%
}

\begin{document}

\lstinputlisting{Program.cs}

\end{document}


Program.cs, that is generated by Visual Studio, is always prefixed with a BOM (Byte Order Mark). If you have no Microsoft Visual Studio installed, you can download a test file named Default.aspx.cs from the official ASP.NET website securely. It is worth mentioning that the backslash in \true is NOT a typo.

The question is how to prevent extendedchars=\true from producing a blank line?

-
@xport: are you sure about \true? Shouldn't it be extendedchars=true without the backslash? –  egreg May 27 '11 at 14:44
@egreg, the backslash is needed. –  xport May 27 '11 at 14:48
and where is the sense of \true instead of true? –  Herbert May 27 '11 at 15:10
@xport: sorry, but \true makes only sense, when it is defined as \def\true{true}. However extendedchars can be used without setting it to true, because true it is the default. –  Herbert May 27 '11 at 15:36
@xport: package listings cannot handle utf8. You have to choose in such a case listingsutf8 –  Herbert May 27 '11 at 15:38

The correct syntax is extendedchars=true without the backslash. But it doesn't make sense without inputenc. And with inputenc and/or fontenc+T1 you don't get the wanted result as then the input will give characters.

If you use a command as value this inserts after the BOM a new line in the code. That's why it looks as if is it a solution to your problem. But actually it will break again if you load inputenc or fontenc.

\documentclass{article}
\usepackage{xcolor}
\usepackage{listings}
\usepackage[T1]{fontenc}
\usepackage[ansinew]{inputenc}

\lstset
{
breaklines=true,
tabsize=3,
showstringspaces=false,
extendedchars=\blub,%<======= SOURCE OF PROBLEM
language={[Sharp]C},
frame=single,
rulecolor=\color{red}%
}

\begin{document}
abc
\lstinputlisting{test-bom.txt}

\end{document}


And here a suggestion how to get around the BOM-problem when your main document is utf8 and the listing is (with the exception of the BOM) pure ASCII:

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{listings}
\usepackage[utf8]{inputenc}

\lstset{%
language={[Sharp]C}}

\begin{document}

\begingroup
\inputencoding{ansinew}
\lstset{
literate={^^ef^^bb^^bf}{}0
}
\lstinputlisting{test-bom.txt}
\endgroup
\end{document}

-
Your code snippet produces 3 "strange" characters at the first row. –  xport May 27 '11 at 16:42
@xport Yes I know. That is what I wanted to demonstrate: The extendedchar solution works only under some conditions. It doesn't work by design but by accident. I have added a suggestion how to solve the BOM problem when the main document is ASCII and the listings contains (beside the BOM) only ASCII chars. If your listings contains non-ASCII-chars it will probably break in any case. –  Ulrike Fischer May 27 '11 at 17:09
This BOM issue only occurs in MikTeX, not in TeXLive. –  kiss my armpit Mar 13 '12 at 14:34

There is no empty line with

\lstset{
breaklines,
tabsize=3,
showstringspaces=false,
extendedchars,
language={[Sharp]C},
frame=single,
rulecolor=\color{red}%
}

-
Using your setting and if you don't include \usepackage[utf8]{inputenc}, there will be no blank line but you will get 3 unnecessary spaces prefixing the first row. –  xport May 27 '11 at 16:21
sure, it is a character and handled by listings, but it was you who said, that you need \true and get an empty line. And as I wrote utf8 is not useful with package listings –  Herbert May 27 '11 at 16:49
Sorry. I mean utf8x in my above comment, not utf8. –  xport May 27 '11 at 17:03
@xport: that doesn't matter –  Herbert May 27 '11 at 17:56

The extendedchars option of the listings package is relevant if you want to typeset source code that includes non-ASCII characters using a TeX engine without native Unicode support such as pdfTeX. Obviously, the listings package has to deal a lot with category codes and, specifically, the extendedchars option addresses collaboration with the inputenc package which traditionally enables non-ASCII encoding of LaTeX input files via character activation.

By current design, the collaboration of the listings package with the inputenc package is restricted to single-byte encodings such as ISO-8859-1 and ISO-8859-15. As a consequence, the default value of the extendedchars option (i.e. true) is no use when the source code to be typeset is encoded in UTF-8, independently of whether the utf8 or the utf8x module is loaded by the inputenc package. In either case, you get errors and faulty output.

If the extendedchars option is set to false (or \true, \chapter, \documentclass etc. ;-) ) the listings package doesn’t attempt to collaborate with the inputenc package. Depending on the combination of active input encoding and UTF-8 character, this may accidentally result in pseudo-correct output (as is the case with the combination utf8x + BOM). In general, however, you get errors and/or faulty output once again.

To sum up: When the source code to be typeset is encoded in UTF-8 playing around with the extendedchars option doesn’t make sense. But there are two other possibilities:

• Resort to the literate option as suggested by Ulrike Fischer.

• Instead of the listings package, use the listingsutf8 package in conjunction with the extended inputencoding option:

...
\usepackage{listingsutf8}
\lstset{%
inputencoding=utf8/ascii,
breaklines=true,
tabsize=3,
showstringspaces=false,
language={[Sharp]C},
frame=single,
rulecolor=\color{red}%
}
...


Actually, the listingsutf8 package is only capable of handling UTF-8 characters that can be translated to some single-byte encoding. UTF-8 characters such as the BOM that can’t be translated to any single-byte encoding are silently ignored. This may or may not be desirable. In case of the BOM, it fortunately is.

-
Have you read my answer: tex.stackexchange.com/a/47816/9467? –  kiss my armpit Mar 13 '12 at 15:28
@DamienWalters: Yes, but if you use pdfTeX and the output seems OK to you it’s just another coincidence. –  mhp Mar 13 '12 at 16:57

This issue only happens in MikTeX. Without using utf8x encoding and/or extendedchars option, this issue disappears if you use TeXLive.

-
If this is the case it’s only by accident. –  mhp Mar 13 '12 at 21:39

Here is the solution that has been actually answered by Vaulty. Unfortunately, I did not realize that \usepackage[utf8x]{inputenc} is needed.

\documentclass{article}
\usepackage{xcolor}
\usepackage[utf8x]{inputenc}%<========= MANDATORY
\usepackage{listings}

\lstset
{
breaklines=true,
tabsize=3,
showstringspaces=false,
extendedchars=\true,%<======= MANDATORY, it is not a typo :)
language={[Sharp]C},
frame=single,
rulecolor=\color{red}%
}

\begin{document}

\lstinputlisting{Default.aspx.cs}

\end{document}

-
again: your setting with \true makes no sense! Same for utf8 –  Herbert May 27 '11 at 16:50
@Herbert: the above code works well. I don't get a blank line. I don't get three strange characters as well. –  xport May 27 '11 at 17:00
@Herbert: Above code uses utf8x. –  xport May 27 '11 at 17:03
sure, as long as your code has no special utf8 character it works, otherwise not! –  Herbert May 27 '11 at 17:55
I’m sorry, I’ve decided to downvote the answer since it only works by accident. Moreover, extendedchars=\true is really misleading (it actually means extendedchars=false). –  mhp Mar 13 '12 at 17:14