\inputencoding{latin1} cannot be used in the user documentation of a .dtx file if it is \DocInput

Third edit: (As pointed out by Ulrike, my problem in fact doesn't concern listingsutf8 but only inputenc, so I reflect this by giving to the question another title and another MWE.)

The following MWE points out that as soon as, in a .dtx file, both:

• the ltxdoc document \DocInputs the .dtx file itself,
• the user documentation (that is, the commented section) selects latin1 encoding with \inputencoding{latin1},

a strange error:

***************************
* Character table correct *
***************************
(/usr/local/texlive/2014/texmf-dist/tex/latex/base/latin1.def

! Package doc Error: Character table corrupted.

See the doc package documentation for explanation.
Type  H <return>  for immediate help.
...

l.53 %%   Right brace   \}     Tilde         \~}

?


is thrown during a pdfLaTeX compilation.

% \iffalse
%<*driver>
\documentclass{ltxdoc}
\usepackage[utf8]{inputenc}
\begin{document}
\DocInput{\jobname.dtx}
\inputencoding{latin1}
Foo bar.
\end{document}
%</driver>
% \fi
%
% \inputencoding{latin1}
%
% \Finale
\endinput


I wonder what's going on.

Note that this strange error doesn't occur if:

1. either \DocInput{\jobname.dtx} is removed,

2. or the \inputencoding{latin1} declaration is directly inserted in the ltxdoc document body and not in the user documentation.

Original edit: The following MWE points out that as soon as, in a .dtx file, both:

• the ltxdoc document:
• loads the listingsutf8 package,
• \DocInputs the .dtx file itself,
• the user documentation contains a lstlisting environment:
• with the listingsutf8's inputencoding=utf8/latin1 option,
• containing utf8 encoded characters,

a strange error:

***************************
* Character table correct *
***************************
(/usr/local/texlive/2014/texmf-dist/tex/latex/base/latin1.def

! Package doc Error: Character table corrupted.

See the doc package documentation for explanation.
Type  H <return>  for immediate help.
...

l.53 %%   Right brace   \}     Tilde         \~}

?


is thrown during a pdfLaTeX compilation.

% \iffalse
%<*driver>
\documentclass{ltxdoc}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{listingsutf8}
\begin{document}
\DocInput{\jobname.dtx}
\begin{lstlisting}
ü
\end{lstlisting}
\end{document}
%</driver>
% \fi
%
% \begin{lstlisting}[inputencoding=utf8/latin1]
% ü
% \end{lstlisting}
%
% \Finale
\endinput


I wonder what's going on.

Note that this strange error doesn't occur if:

1. either \DocInput{\jobname.dtx} is removed,
2. or inputencoding=utf8/latin1 option of lstlisting environment is removed (in this case, a normal Package inputenc Error: Unicode char occurs),
1. the lstlisting with inputencoding=utf8/latin1 option and containing utf8 encoded characters is directly inserted in the ltxdoc document body,

2. \usepackage[T1]{fontenc} and \usepackage[utf8]{inputenc} are removed, listingsutf8 is replaced by listings, and XeLaTeX or LuaLaTeX compilation are used.

Second edit: The same problem arises with a more conform use of listingsutf8:

% mylistings.txt file
\begin{lstlisting}
ü
\end{lstlisting}


and:

% .dtx file
% \iffalse
%<*driver>
\documentclass{ltxdoc}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{listingsutf8}
\begin{document}
\DocInput{\jobname.dtx}
\lstinputlisting[inputencoding=utf8/latin1]{mylisting.txt}
\end{document}
%</driver>
% \fi
%
% \lstinputlisting[inputencoding=utf8/latin1]{mylisting.txt}
%
% \Finale
\endinput

• Did you use the environment only for the example (inputencoding=utf8/latin1 works only with the \lstinputlisting). The source of the error is imho that latin1.def is read in with \endlinechar=-1 (or some other relevant change) and then the comparision of the character tables fails. – Ulrike Fischer Mar 12 '15 at 20:17
• Indeed, I came from a more complex situation (involving tcolorbox's inputencoding=utf8/latin1 option) and forgot meanwhile listingsutf8 claims "[o]nly \lstinputlisting is supported by the syntax extension of key inputencoding". But (1) a MWE with \documentclass{article}, \usepackage{listingsutf8} and a lstlisting environment with inputencoding=utf8/latin1 option and containing utf8 encoded characters works like a charm, (2) the same problem as described arises with \lstinputlisting. Do you think the bug should be reported to inputenc maintainers? – Denis Bitouzé Mar 12 '15 at 21:06
• You can report it, but I don't know if it counts as bug and if something could be done. Btw: you can remove all the listings stuff, the error is triggered by \inputencoding{latin1} in the commented section. – Ulrike Fischer Mar 12 '15 at 22:01
• I'll report the bug. You're right, the MWE can be made still more minimal :) and I'll edit my question in this respect. – Denis Bitouzé Mar 13 '15 at 6:05
• @DenisBitouzé I have made a LaTeX bug report with id latex/4420 and also proposed the solution of my answer for fixing the issue. – Heiko Oberdiek Mar 17 '15 at 2:47

The wrong character table comes from the wrong setting for \endlinechar, see Ulrike's comment. The setting of \endlinechar to -1 is done in \inputencoding to prevent additional spaces by line ends and \par tokens by empty lines, if \inputencoding is used in horizontal mode. Thus this setting of \endlinechar is necessary, not a bug.

Thus the question is rather, why is the character table in the comments is checked in the first place. latin1.def is not the argument of \DocInput. The reason is the changed category code of the percent sign, which is active and not a comment character. Therefore \inputencoding can be made more robust, by ensuring the category code of the percent character.

Minimal example with applied fix:

% \iffalse
%<*driver>
\documentclass{ltxdoc}
\usepackage[utf8]{inputenc}

\usepackage{etoolbox}
\makeatother
\pretocmd\inputencoding{%
\xdef\saved@percent@catcode{\the\catcode\%}%
\catcode\%14\relax
}{}{%
\errmessage{Cannot patch \string\inputencoding}%
}
\apptocmd\inputencoding{%
\catcode\%\saved@percent@catcode\relax
}{}{%
\errmessage{Cannot patch \string\inputencoding}%
}
\makeatother

\begin{document}
\DocInput{\jobname.dtx}
\end{document}
%</driver>
% \fi
%
% Hello\inputencoding{latin1} World
%
% \Finale
\endinput


If LaTeX3 category codes are in force (\ExplSyntaxOn, or implicitly by \ProvidesExplPackage, see comment), they have to be disabled, because \inputencoding is not encoded by expl3 macros:

\RequirePackage{expl3}
\RequireExplPackage{...}
...
\ExplSyntaxOff
\pretocmd ...
\apptocmd ...
\ExplSyntaxOn


Other ways of patching:

\newcommand{\org@inputencoding}{}
\let\org@inputencoding\inputencoding
\def\inputencoding#1{%
\xdef\saved@percent@catcode{\the\catcode\%}%
\catcode\%14\relax
\org@inputencoding{#1}%
\catcode\%\saved@percent@catcode\relax
}


Or more elegant, package inputenc provides two hooks, that are called at the beginning and end of \inputencoding:

\inpenc@prehook=\expandafter{\the\inpenc@prehook
\xdef\saved@percent@catcode{\the\catcode\%}%
\catcode\%14\relax
}
\inpenc@posthook=\expandafter{\the\inpenc@posthook
\catcode\%\saved@percent@catcode\relax
}

• Unfortunately, this patch doesn't work in case it is performed in a LaTeX3 package, as pointed out by this minimal example. – Denis Bitouzé Mar 17 '15 at 8:03
• @DenisBitouzé I have added two additional ways for patching, which both work with package expl3. – Heiko Oberdiek Mar 17 '15 at 8:59
• @DenisBitouzé In an expl3` package, the category codes are different, see updated answer. – Heiko Oberdiek Mar 17 '15 at 9:48
• That works, indeed :) – Denis Bitouzé Mar 17 '15 at 12:05
• This issue has been fixed in the upcoming LaTeX release – Frank Mittelbach Mar 28 '15 at 11:46