3

I would like to have an expandable macro that extracts the first (and sometimes the second) character of UTF-8/Cyrillic text strings without using additional packages. No simple solutions from TeX or LaTeX work with UTF-8/Cyrillic strings.

I give below an example of a working macro, which is partially taken from Get the first and second character of a macro argument :

\documentclass{article}

\usepackage[T2A]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[russian]{babel}

\makeatletter
\newcommand{\firstof}[1]{\@car#1\@nil}
\makeatother

\begin{document}

\firstof{Vladimir}

\end{document}

Unfortunately, this example fails with the error Error: Invalid UTF-8 byte sequence (Ð\par) using Cyrillic strings like \firstof{Владимир}.

I roughly understand that by default TeX is not adapted to manipulating strings with multibyte characters, but this problem is solved in some packages. However, I do not want to use other packages for such a simple problem (as it seems at first glance) and I will be grateful to the community for help and tips.

Ideally, I would like to have an expandable macro like \newcommand{\firstof}[2][1]{.....}, which by default for UTF-8/Cyrillic strings returns the value of the first character, for example, in the case of \firstof{Владимир} returns В, and for \firstof[2]{Владимир} returns Вл, and these chars could be used in /ifx to compare with others and written to a file using \write.

7
  • 1
    Why "without additional package"? Remember that package is just other people's code, so you can just copy their code and include in yours (if the license permit).
    – user202729
    Jan 28, 2022 at 22:37
  • 1
    • Just use char_to_utfviii_bytes:n from expl3. You can learn expl3, right? • "simple problem" remember that TeX's built in functionality is very limited. and without expl3 you need to do massive data juggling to get useful computation. • It looks like you're not using LuaTeX, want to give it a try? Programming in Lua is much simpler.
    – user202729
    Jan 28, 2022 at 22:37
  • Getting the first or second character is doable, but you won't be able to compare Вл with an \ifx (and get the result you expect) because each of those characters is two tokens (thus four tokens in total) and \ifx only compares two tokens at a time. You could compare with \pdfstrcmp though Jan 29, 2022 at 1:49
  • @user202729 Unfortunately, I'm not familiar with LaTeX3/expl3 syntax, but I could probably learn it. As it turned out, the only package for PdfLaTeX that can correctly extract Cyrillic and ASCII characters from strings is xstring. But unfortunately its commands are not expandable and this then causes a lot of problems when concatenating author name strings in loops and then writing them to a file.
    – Crosfield
    Jan 29, 2022 at 9:15
  • @user202729 And also I've only heard about the Lua language, but I'm not familiar with it. My programming knowledge ended many years ago in the Pascal.
    – Crosfield
    Jan 29, 2022 at 9:22

3 Answers 3

2

Each byte of the UTF-8 encoding is a separate token in pdflatex, however you can recognise the leading token which tells you how many bytes are needed. This version covers the one and two byte cases.

enter image description here

\documentclass{article}

\usepackage[T2A]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[russian]{babel}

\makeatletter
\newcommand{\firstof}[1]{\expandafter\checkfirst#1\@nil}
\def\checkfirst#1{%
  \ifx\UTFviii@two@octets#1%
  \expandafter\gettwooctets
  \else
  \expandafter\@car\expandafter#1%
  \fi
}
\def\gettwooctets#1#2#3\@nil{\UTFviii@two@octets#1#2}

\makeatother

\begin{document}

\firstof{Vladimir}

\firstof{Владимир}

\end{document}

If you want to handle the rest of the input as opposed to discarding everything after the first letter, you can make a small change so that you pass in a command to appy to the remaning text. If you pass in \gobble it extracts as before. If you pass in \firstofx\gobble then it exctracts the first letter of the remaining text so you get two letters:

enter image description here

\documentclass{article}

\usepackage[T2A]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[russian]{babel}

\makeatletter
\newcommand{\firstofx}[2]{\expandafter\checkfirst#2\@nil{#1}}
\def\checkfirst#1{%
  \ifx\UTFviii@two@octets#1%
  \expandafter\gettwooctetsx
  \else
  \expandafter\getasciix\expandafter#1%
  \fi
}

\def\getasciix#1#2\@nil#3{#1#3{#2}}

\def\gettwooctetsx#1#2#3\@nil#4{\UTFviii@two@octets#1#2#4{#3}}

\newcommand\gobble[1]{}

\makeatother

\begin{document}

\firstofx\gobble{Vladimir}

\firstofx{\firstofx\gobble}{Vladimir}

\firstofx\gobble{Владимир}

\firstofx{\firstofx\gobble}{Владимир}


\end{document}
11
  • WOW! Thanks, David for the elegant, nice and short solution!
    – Crosfield
    Jan 29, 2022 at 13:21
  • @Crosfield Except that there might be some indeterminate amount of code hidden behind \UTFviii@two@octets... if it's library code it doesn't matter if it's long or short anyway. (okay, maybe not that large, just one or two constants to compare with)
    – user202729
    Jan 29, 2022 at 13:43
  • @user202729 not sure what you mean? \UTFviii@two@octets is a core latex command it has a known definition and so I'm not sure what you mean by indeterminate? (actually its expansion isn't consulted here so it doesn't really matter what its definition is, just that the first byte of two byte characters have a definition that starts this way. Jan 29, 2022 at 13:56
  • @DavidCarlisle That's only with regard to the comment above... what I mean is, despite the short code, the logic might be complex (as complex as the other solutions) if the function is implemented internally with lots of code.
    – user202729
    Jan 29, 2022 at 14:07
  • @DavidCarlisle Did I understand correctly that to get the first and second character I should use \UTFviii@four@octets and for this I should use the functions like \newcommand{\firstsecondof}[1]{\expandafter\checkfirstsecond#2\@nil} \def\checkfirstsecond#1{% \ifx\UTFviii@four@octets#1\expandafter\getfouroctets \else\expandafter\@car#1\@nil\@car\@cdr#1\@nil\@nil\fi \def\getfouroctets#1#2#3#4#5\@nil{\UTFviii@four@octets#1#2#3#4}
    – Crosfield
    Jan 29, 2022 at 14:15
3

This should probably work. I defined \headof and \tailof (using code stolen from here and here), which do what their names promise. If you use \headof* (or \tailof*), it will expand its argument, so you can effortlessly get the nth character out of a sequence by nesting \headof* and \tailof* (pretty much like car and cdr in lisp, if you're into that). For example, the fourth character of Владимир can be extracted with

\headof*{\tailof*{\tailof*{\tailof{Владимир}}}}

Annoyingly easy :)

You can then compare strings using the primitive \pdfstrcmp or a higher level \str_if_eq:eeTF. \ifx won't work because it compares two tokens, and л (for example) is two tokens by itself (assuming you're using pdfTeX, of course).

If the argument is empty, the result is also empty. If the head is a group of tokens (within {...}), the group is treated as a single thing and returned without the outer braces.

\documentclass{article}

\usepackage[T2A]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[russian]{babel}

\ExplSyntaxOn
\NewExpandableDocumentCommand \headof { s +m }
  { \IfBooleanT {#1} { \exp_args:Ne } \crosfield_text_head:n {#2} }
\NewExpandableDocumentCommand \tailof { s +m }
  { \IfBooleanT {#1} { \exp_args:Ne } \crosfield_text_tail:n {#2} }
\cs_new:Npn \crosfield_text_head:n #1
  {
    \tl_if_head_is_N_type:nTF {#1}
      { \__crosfield_get_head:Nw #1 \q_stop }
      {
        \tl_if_head_is_group:nTF {#1}
          { \exp_not:o { \use_i_delimit_by_q_stop:nw #1 \q_stop } }
          { \tl_if_empty:nTF {#1} { } { ~ } }
      }
  }
\cs_new:Npn \crosfield_text_tail:n #1
  {
    \tl_if_head_is_N_type:nTF {#1}
      { \__crosfield_get_tail:Nw #1 \q_stop }
      {
        \tl_if_head_is_group:nTF {#1}
          { \exp_not:o { \use_none:n #1 } }
          { \tl_if_empty:nTF {#1} { } { \exp_not:o { \exp:w \exp_end_continue_f:w #1 } } }
      }
  }
\bool_lazy_or:nnTF
    { \sys_if_engine_luatex_p: }
    { \sys_if_engine_xetex_p: }
  {
    \cs_new:Npn \__crosfield_get_head:Nw #1 #2 \q_stop { \exp_not:N #1 }
    \cs_new:Npn \__crosfield_get_tail:Nw #1 #2 \q_stop { \exp_not:n {#2} }
    \use_none:n
  }
  { \makeatletter \use:n }
  {
    \makeatother
    \cs_new:Npn \__crosfield_get_head:Nw
      { \__crosfield_head_tail:NNw \use_i:nn }
    \cs_new:Npn \__crosfield_get_tail:Nw
      { \__crosfield_head_tail:NNw \use_ii:nn }
    \cs_new:Npn \__crosfield_head_tail:NNw #1 #2 #3 \q_stop
      {
        \use:e
          {
            \exp_not:N \__crosfield_head_tail:w
              \exp_not:o { \token_to_meaning:N #2 }
              \tl_to_str:n { UTFviii@ one @octets } ~
          }   \q_stop { #2 #3 } #1
      }
    \use:e
      {
        \cs_new:Npn \exp_not:N \__crosfield_head_tail:w
          #1 \tl_to_str:n { UTFviii@ } #2 \tl_to_str:n { @octets } ~ #3
          \exp_not:N \q_stop #4 #5
      }
      {
        \str_case:nnTF {#2}
          {
            { one   } { \__crosfield_head_or_tail:NNw #5 }
            { two   } { \__crosfield_head_or_tail:NNNw #5 }
            { three } { \__crosfield_head_or_tail:NNNNw #5 }
            { four  } { \__crosfield_head_or_tail:NNNNNw #5 }
          }
          { #4 \q_stop }
          { \ERROR? }
      }
    \cs_new:Npn \__crosfield_head_or_tail:NNw #1 #2 #3 \q_stop
      { \exp_not:o { #1 {#2} {#3} } }
    \cs_new:Npn \__crosfield_head_or_tail:NNNw #1 #2#3 #4 \q_stop
      { \exp_not:o { #1 {#2#3} {#4} } }
    \cs_new:Npn \__crosfield_head_or_tail:NNNNw #1 #2#3#4 #5 \q_stop
      { \exp_not:o { #1 {#2#3#4} {#5} } }
    \cs_new:Npn \__crosfield_head_or_tail:NNNNNw #1 #2#3#4#5 #6 \q_stop
      { \exp_not:o { #1 {#2#3#4#5} {#6} } }
  }
\cs_new_eq:NN \StrCompare \str_if_eq:eeTF
\ExplSyntaxOff

\begin{document}

\StrCompare
  {\headof*{\tailof{Владимир}}}{\detokenize{л}}
  {\true}{\false}

\end{document}
7
  • Thanks Phelype for the reply! I compiled your example. The \StrCompare function really works and correctly compares the values, and maybe I misunderstood something, but the \headof{Владимир} function does not output anything in PDF (unlike \headof{Vladimir}, which outputs V).
    – Crosfield
    Jan 29, 2022 at 9:03
  • @Crosfield You will need a font with Cyrillic glyphs to display Cyrillic: either, for pdflatex, use a legacy T2A encoding font; or use fontspec and a unicode/OpenType font like in Mico's answer, for xelatex/lualatex.
    – Cicada
    Jan 29, 2022 at 11:13
  • @Cicada Thanks for the advice, but I'm using fonts with cyrillic characters. Commands in the preamble \usepackage[T2A]{fontenc} \usepackage[utf8]{inputenc} \usepackage[russian]{babel} allow Cyrillic fonts to be used. Any Cyrillic text inserted after \begin{document} will be output to PDF with this settings and pdflatex engine.
    – Crosfield
    Jan 29, 2022 at 11:55
  • @Crosfield I don't know what to tell you (besides what Cicada said). If I take the document in my answer and write \headof*{\tailof{Владимир}}, I get л in the pdf (also with just \headof{Владимир}, but В looks an awful lot like an ASCII B, so I tested with something else :) Jan 30, 2022 at 1:17
  • @PhelypeOleinik It was probably some kind of glitch. I'm using macOS with MacTeX and TeXShop which is a little bit buggy in the latest version. I remember exactly that yesterday your example worked halfway only (I described the problems that were yesterday). Today I opened it - everything is ok! :) Thanks, Phelype! I understand correctly, the asterisk version is used in cases where there are nested \headof or \tailof commands?
    – Crosfield
    Jan 30, 2022 at 10:15
0

Here's a LuaLaTeX-based solution. The solution doesn't employ the standard string.sub Lua function, at it isn't unicode-aware. Instead, it employs the unicode.utf8.sub function to extract the first n (by default, n=1) characters of the string argument. The macro \firstof is expandable.

enter image description here

% !TEX TS-program = lualatex
\documentclass{article}
\usepackage[russian]{babel}
\usepackage{fontspec}
\setmainfont{Noto Serif} % select a suitable opentype text font

\newcommand\firstof[2][1]{\directlua{%
    tex.sprint ( unicode.utf8.sub ( "#2" , 1 , #1 ) ) }}

\newcommand\zzz{Владимир} 

\begin{document}
\firstof{\zzz}, \firstof[2]{\zzz}, \firstof[4]{\zzz}
\end{document}
2
  • In analogy to \firstof, one could define \lastof like this: \newcommand\lastof[2][1]{\directlua{ tex.sprint ( unicode.utf8.sub ( "#2" , -#1 ) ) }}.
    – Mico
    Jan 29, 2022 at 5:27
  • Thanks for the help. I will try to look at the abilitiy of the Lua language
    – Crosfield
    Jan 31, 2022 at 9:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .