0

I have a problem getting a macro working with UTF-8 characters. I was able to boil down my problem to the following non-working minimal example:

\documentclass{beamer}
\def\ta#1#2{BA}
\begin{document}
\begin{frame}

\ta1 ä
\end{frame}
\end{document}

Which produces a Invalid UTF-8 byte "A4 error. My file is in UTF-8 encoding.

Looking at the UTF-8 table it is obvious where the A4 comes from, it is the second byte of the umlaut.

However, I have no idea how to fix this.

How would I properly design a macro which is supposed to pick up individual letters - without splitting UTF characters into several bytes??

Update: Learned a lot here but still stuck with the ultimate use case, which I shall copy in an example of:

\documentclass{article}
\usepackage{tabto}

\makeatletter
\def\tb#1#2{\def\ca{#1}\expandafter\tba#2}

\def\tba#1{%
\ifx#1+\tabto{\dimexpr\ca cm + 0.5cm}%
\else\ifx#1-\tabto{\dimexpr\ca cm - 0.5cm}%
\else\tabto{\dimexpr\ca cm}%
\ifx\UTFviii@two@octets#1\expandafter\tbaa\fi%
\ifx\UTFviii@three@octets#1\expandafter\tbab\fi%
\ifx\UTFviii@four@octets#1\expandafter\tbac\fi%
\fi\fi%
\relax#1%
}

\def\tbaa#1#2{#1#2}
\def\tbab#1#2#3{#1#2#3}
\def\tbac#1#2#3#4{#1#2#3#4}

\begin{document}

VOR\tb3- Ü

VOR \tb3 Ü

VOR \tb3+ Ü
\end{document}

I cannot get rid of the printout of the + and - in the + and - case. Whatever I tried always produces yet another UTF-8 coding error.

8
  • 1
    pdftex is an 8bit program, unicode characters are not single units they are simply the utf8 byte sequence so you need to pick up the first byte that will tell you how many bytes are in that character's utf-8 encoding, then you can collect all the needed bytes. Commented Nov 23, 2020 at 23:25
  • 1
    or simpler, if it is suitable in the context, insist on fully braced arguments,\ta{1}{ä} would work fine. Commented Nov 23, 2020 at 23:29
  • I expected so but had not point to start on how to parse the individual bytes. Your answer below provides a very nice start for me. Thanx a lot. Commented Nov 24, 2020 at 9:52
  • To save on typing efforts I rather have no blanks and no braces. The original use case needs to distinguish \tabu2 and \tabu2+ and \tabu2- which should all be different, parameter-free macros for tabulation. \tabu2 being \tabto{2cm} \tabu2- being \tabto{1.5cm} and \tabu2+ begin \tabto{2.5cm}. I implemented that as nested if's starting with \tabu. Commented Nov 24, 2020 at 9:55
  • 2
    As I show below it is possible but using braces is far simpler and more robust. Even if your editor shows you ä as a single character, to pdftex it really isn't. It is the two character C3 and A4, you could even write it as \ta1 ^^c3^^a4 So whenever you use any macro with unbraced arguments you will always get individual bytes like this, compare \fbox{a}\fbox a to \fbox{ä} \fbox ä Omitting braces is not "a convenience to save typing" it is "a route to unending pain" Commented Nov 24, 2020 at 10:42

1 Answer 1

3

You need to brace the arguments to keep things together (your \ta macro works unchanged with braced arguments, although I add a \typeout here for debugging. Or you need to inspect the first byte, and then collect as many bytes as are in that character's UTF-8 encoding (\tb here)

This produces a log

1,ä
macro:->1,macro:->\UTFviii@two@octets ä
2,𝐀
macro:->2,macro:->\UTFviii@four@octets 𝐀

from

\documentclass{beamer}
\def\ta#1#2{%
\typeout{\detokenize{#1},\detokenize{#2}}%
BA}

\makeatletter
\def\tb#1#2{%
\def\ca{#1}\expandafter\tba#2}
\def\tba#1{%
\ifx\UTFviii@two@octets#1\expandafter\tbaa\fi
\ifx\UTFviii@three@octets#1\expandafter\tbab\fi
\ifx\UTFviii@four@octets#1\expandafter\tbac\fi
\relax#1}
\def\tbaa#1\relax#2#3#4{%
\def\cb{#2#3#4}%
\typeout{\meaning\ca,\meaning\cb}%
BA}
\def\tbab#1\relax#2#3#4#5{%
\def\cb{#2#3#4#5}%
\typeout{\meaning\ca,\meaning\cb}%
BA}
\def\tbac#1\relax#2#3#4#5#6{%
\def\cb{#2#3#4#5#6}%
\typeout{\meaning\ca,\meaning\cb}%
BA}

\begin{document}
\begin{frame}

\ta{1}{ä}

\tb 1 ä

\ta{2}{𝐀}

\tb 2 𝐀


\end{frame}
\end{document}
2
  • I am studying your solution, trying to adapt it to my ultimate use case. I am stuck with understanding the role of parameter #1 and #4 in tbaa. If we face a two-octet UTF, we have to take care of tow tokens. But why do we need the #1 relax and why do we need #4 when we are in a case of only two octets in tbaa? I'll copy in my utlimate use case into the question. I still have to get rid of the printout of the + and - and do not manage to see how. All things I try always deliver UTF-8 error or unwanted results. :-( Commented Nov 24, 2020 at 12:49
  • you can add \tracingall to see it in action and the value of each parameter but #1\relax is to gather up and discard all the other cases, so in the two octet case you need to lose the tests for the three and four octet cases so everything up to \relax is gobbled as #1 and discarded. Commented Nov 24, 2020 at 13:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .