Parsing utf8 problems

Question

There are a number of questions relating to problems when using utf8 input encoding, and the closest I found to my problem is, I think, What characters in a normal text document will screw up LaTeX?.

A user of my censor package asked if I can get the package's \blackout macro to work with utf8 umlauts. So far, the answer is "no". I narrowed the problem down to the macro \bl@t which censors argument #2 and then reinvokes a recursion via argument #1. The problem, best I can tell, is that utf8 encoding requires more than 1 byte for things like umlauted characters, and so the #2 passed to \bl@t is only half a character, and so it chokes.

Here is an MWE that, if you uncomment either of the two commented lines, will break the code:

\documentclass{article}
\usepackage[ngerman]{babel}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{censor}

\makeatletter
\def\stringend{$}

\long\def\blackout#1{\def~{-}\censor@Block#1\stringend\let~\sv@tilde}
\long\def\censor@Block{\IfNextToken\stringend{\@gobble}%
  {\IfNextToken\@sptoken{ \bl@t{\censor@Block}}%
  {\bl@t{\censor@Block}}}}

\long\def\bl@t#1#2{\if\bpar#2\par\else\if.#2\censordot\else\censor{#2}\fi\fi#1}
\makeatother

\begin{document}
äöüß, \censor{äöüß}\par
\blackout{ab\par cd}\par
%\blackout{ä}\par

\makeatletter
%\bl@t xä
\end{document}

The package's \censor macro works fine on the umlauted stuff, but \blackout and more specifically, the \bl@t service routine, do not. If you want it more simplified, you can think of \bl@t as \def\bl@t#1#2{\censor{#2}#1} (but this will not work with \pars in the input stream). The #1 is always a reinvocation of \censor@Block on the remaining input string.

---

EDIT: It would seem that, if a multi-byte input character is next in the input stream, then this definition

\long\def\bl@t#1#2#3{\if\bpar#2#3\par\else\if.#2#3\censordot\else
       \censor{#2#3}\fi\fi#1}

can absorb it properly. Thus, reversing which invocations are commented:

%\blackout{ab\par cd}\par
\blackout{äöüß}\par

\makeatletter
\bl@t xä

works fine. So the key will be to be able to determine in advance which type of character lies next in the input stream and choose the appropriate parsing method.

Answer 1

rather than worry about utf-8 it's probably best to let inputenc worry about that and expand all characters to LICR (latex internal form) then you would also be able to support [latin1] or anything else. I also changed \if it \ifx as \if looked wrong (although I don't really know what the code is doing:-)

\documentclass{article}
\usepackage[ngerman]{babel}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{censor}

\makeatletter
\def\stringend{$}


\long\def\blackout#1{%
\protected@edef\tmp{#1}%
\def~{-}\expandafter\censor@Block\tmp\stringend\let~\sv@tilde}
\long\def\censor@Block{\IfNextToken\stringend{\@gobble}%
  {\IfNextToken\@sptoken{ \bl@t{\censor@Block}}%
  {\bl@t{\censor@Block}}}}

\long\def\bl@t#1#2{%
\ifx\bpar#2\let\next\par\else\ifx.#2\let\next\censordot\else\def\next{\censor{#2}}\fi\fi
  \next#1}

\makeatother

\begin{document}
äöüß, \censor{äöüß}\par
abcd, \censor{abcd}\par
\blackout{ab\par cd}\par
\blackout{ä}\par


\end{document}