6

I'm currently using expl3 code in a project. It's been a slow learning process.

Now I need to deal with an UTF-8 input stream, but, as I already suspected there's a problem dealing with 2-byte encodings.

Actually, if the input is UTF-8 or not is not relevant for my case. I want to process, for example até and have the following outcome: a, t, 0xC3 and 0xA9. In other words, processing it one byte at a time. In LaTex, I want to have something like \processtl{até} and have at{0xC3}{0xA9} as the 4-token tl.

\documentclass[12pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[brazilian]{babel}

\ExplSyntaxOn

\newcommand{\processtl}[1]{
    \tl_set:Nn \l_tokens_tl { #1 }
    \tl_map_inline:Nn \l_tokens_tl { (##1) } %% Here: check one byte!
}

\ExplSyntaxOff

\begin{document}
token list: \processtl{abc{$\beta$}}\par
% token list with á and ç: \processtl{ábç{$\beta$}}\par  % FAILS
\end{document}

Is there a way to recognize the 0xC3 and treat it separately from the following byte?

Something like:

\IfTestByte{#1=0xC3){                  % check if current byte is the first UTF-8
    \showinhexadecimal(0xC3}           % output it in hex
    \getnexttoken                      % get the second byte
    \showinhexadecimal{\thenexttoken}  % output it in hex
}

I can't convert tl to str because some of the tokens can be macros and type str detokenize them.

I took a look at the char functions, but coudn't find any to help. Did I miss something useful?

2
  • 1
    if you use pdftex (rather than luatex or xetex) then the tokens are always single byte representing (in this case) the UTF-8 stream, you can not have a single token for the characters past 7F so it isn't clear what you are asking. Dec 14, 2021 at 22:20
  • 1
    \char_to_utfviii_bytes:n perhaps?
    – Joseph Wright
    Dec 14, 2021 at 22:27

2 Answers 2

5

With pdflatex the function \tl_map_inline:nn will just consider á as two bytes. We can check whether the character code is a high byte and act consequently. If the object passed to it is more than one token, like for {$\beta$}, we can print it as is.

\documentclass[12pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[brazilian]{babel}

\ExplSyntaxOn

\NewDocumentCommand{\processtl}{m}
 {
  \jander_process_tl:n { #1 }
 }

\cs_new_protected:Nn \jander_process_tl:n
 {
  \tl_map_inline:nn { #1 }
   {
    ( % open parenthesis
    \tl_if_single:nTF { ##1 }
     {
      \__jander_process_highbyte:en { `\token_to_str:N ##1 } { ##1 }
     }
     { ##1 }
     ) % close parenthesis
   }
 }
\cs_new:Nn \__jander_process_highbyte:nn
 {
  \int_compare:nTF { #1>127 } { 0x\int_to_Hex:n { #1 } } { #2 }
 }
\cs_generate_variant:Nn \__jander_process_highbyte:nn { e }

\ExplSyntaxOff

\begin{document}

token list: \processtl{abc{$\beta$}}

token list with á and ç: \processtl{ábç{$\beta$}}

\end{document}

What happens? When á is found, it's actually 0xC3 followed by 0xA1. In either case it's a sequence of single tokens, as far as pdftex is concerned. So we can compute its character code (but we need to make it an “other character”, because it might be active) and pass it to another function that checks whether it's a high byte.

enter image description here

A modification for taking into account that a single token might be a control sequence.

\documentclass[a4paper]{article}

\ExplSyntaxOn

\NewDocumentCommand{\processtl}{m}
 {
  \sys_if_engine_pdftex:TF
   {
    \jander_process_tl_onebyte:n { #1 }
   }
   {
    \jander_process_tl_unicode:n { #1 }
   }
 }

\cs_new_protected:Nn \jander_process_tl_onebyte:n
 {
  \tl_map_inline:nn { #1 }
   {
    ( % open parenthesis
    \bool_lazy_or:nnTF { ! \tl_if_single_p:n { ##1 } } { \token_if_cs_p:N ##1 }
     {
      ##1
     }
     {
      \__jander_process_highbyte:en { `\token_to_str:N ##1 } { ##1 }
     }
     ) % close parenthesis
   }
 }
\cs_new:Nn \__jander_process_highbyte:nn
 {
  \int_compare:nTF { #1>127 } { 0x\int_to_Hex:n { #1 } } { #2 }
 }
\cs_generate_variant:Nn \__jander_process_highbyte:nn { e }

\ExplSyntaxOff

\begin{document}

token list: \processtl{abc{$\beta$}}

token list with á and ç: \processtl{ábç{$\beta$}\LaTeX}

\end{document}

enter image description here

The trick to save coding is to use \bool_lazy_or:nnTF; the first condition reads “not a single token” and the second “a control sequence”. Since we use lazy evaluation, when the first condition is true (so we don't have a single token) the second is not evaluated.

enter image description here

4
  • That's very interesting. I should have thought about converting a single token to string! Great answer!
    – Jander
    Dec 15, 2021 at 12:33
  • Is it possible to check if a token is a macro (e.g. \something). For example, if I \def\myalpha{$\alpha$} and try \processtl{abc{$\beta$}\myalpha} it fails with "Undefined control sequence. ...n list: \processtl{abc{$\beta$}{\myalpha}}". I believe this happens because \myalpha is a single token and the code tries to get it's hex value.
    – Jander
    Dec 15, 2021 at 17:53
  • Maybe with \cs_if_exist_p:N?
    – Jander
    Dec 15, 2021 at 17:59
  • @Jander No, it's \token_if_cs:NTF, but we can do better and use “lazy or evaluation“. See update.
    – egreg
    Dec 15, 2021 at 18:08
5

I think you want something like this (it could print in hex with a bit more effort)

enter image description here

\documentclass[12pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[brazilian]{babel}

\ExplSyntaxOn
\makeatletter

\newcommand{\processtl}[1]{{
\def\UTFviii@two@octets##1{\number\expandafter`\string##1}
\def\UTFviii@invalid@err##1{\number\expandafter`\string##1}
    \tl_map_inline:nn {#1} { (##1) } %% Here: check one byte!
}}
\makeatother
\ExplSyntaxOff

\begin{document}
token list: \processtl{abc{$\beta$}}

 token list with á and ç: \processtl{ábç{$\beta$}}\par  % FAILS
\end{document}
2
  • A very compact answer! I have to take a closer look at it since I just don't get it. Where can I find more information related to \UTFviii@two@octets?
    – Jander
    Dec 15, 2021 at 12:43
  • it's the handling of UTF-8 in inputenc (so pre-loaded into the format in current releases) see texdoc utf8ienc.pdf which should show you utf8ienc.pdf from latex base distribution. @Jander Of course for the full range you should have (identical) lines for \UTFviii@three@octets and \UTFviii@four@octets Dec 15, 2021 at 12:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .