6

When I use the macro \StrLen on a string that contains one or more "exotic" characters, the value returned does not correspond to the actual number of characters in the string.

I suspect this is because some of these characters use more than 1 byte for encoding. That being said, is there a trick to getting the "real" length of a string?

\StrLen{aàáâãäåāăąæ} % returns 57 for 11 characters
4
  • I have tried two ways : 1. in the preamble, as a test command like \NewDocumentCommand\mycmd{}{\StrLen{aàáâãäåāăąæ}} 2. between \begin{document} and \end{document}, as a simple statement, \StrLen{aàáâãäåāăąæ}. But neither returns a "good" character count. Jun 6, 2022 at 8:13
  • It would be useful to know what you are hoping to get out of this. For some accented characters there may be a single Unicode codepoint, but there are places that won't be the case
    – Joseph Wright
    Jun 6, 2022 at 8:14
  • Well, what I would like to obtain is the total character count in a string, for intervening on each character of it, in a \loop mode ... Jun 6, 2022 at 8:21
  • 5
    there is no single or simple definition of "character" here and the number you get depends on the definition that you use. Jun 6, 2022 at 8:45

4 Answers 4

11

In UTF-8, non-ASCII characters take several bytes, and xstring's \StrLen might be counting these bytes... however, it seems to be something entirely different, as @SandyG noted.

The following example shows how to count Unicode code points using expl3 functions in a given string, and how to iterate over them. It should work with any engine (recent enough for expl3).

\documentclass{article}
% Font setup: only needed for typesetting the sample string
\usepackage{iftex}
\ifpdftex
  \usepackage{lmodern}
  \usepackage[T1]{fontenc}
\fi

\ExplSyntaxOn
\NewDocumentCommand \setToNbCodePoints { O{default} m m }
  {
    \str_set_convert:Nnnn \l_tmpa_clist {#3} {#1} { clist }
    \cs_set:Npx #2 { \clist_count:N \l_tmpa_clist }
  }

\NewDocumentCommand \iterateOverCodePoints { O{default} m m }
  {
    \str_set_convert:Nnnn \l_tmpa_clist {#2} {#1} { clist }
    \clist_map_inline:Nn \l_tmpa_clist {#3}
  }

\NewDocumentCommand \printCodePoints { O{default} m }
  {
    \str_set_convert:Nnnn \l_tmpa_clist {#2} {#1} { clist }
    \clist_use:Nnnn \l_tmpa_clist { ~and~ } { ,~ } { ,~and~ }
  }
\ExplSyntaxOff

\begin{document}

\setToNbCodePoints{\result}{aàáâãäåāăąæ}%
There are \result~code points in “aàáâãäåāăąæ”. These are:
\iterateOverCodePoints{aàáâãäåāăąæ}{#1, } and that's it.

Put in another way: \printCodePoints{aàáâãäåāăąæ}.

\end{document}

enter image description here

Note: as Joseph Wright hinted, in many cases, there is only one code point per “character”, but this is not always true. For instance, an é can be written using two Unicode code points (one for e, one for the accent): this is what is called combining characters (and causes problems in some software).

The notion of “character” is not well-defined in general: I suggest reading Glyphs, graphemes and other Unicode species in UTF-8 everywhere.

Example with combining characters

Combining an n with U+0308 COMBINING DIAERESIS, which gives (compile with LuaTeX or XeTeX):

\setToNbCodePoints{\result}{n̈}%
Example with two combining characters: there are \result~code points in “n̈”:
\printCodePoints{n̈}.

you'll see:

enter image description here

Note: the code run under pdfTeX can also count these code points, work on them but not typeset this particular combination:

Unicode character ̈ (U+0308) not set up for use with LaTeX.

Even an built from e and U+0301 COMBINING ACUTE ACCENT, which could be represented in the T1 encoding, causes this kind of error when typesetting.

Typesetting from a list of code points

Going in the reverse direction in order to typeset glyphs built from the code points that were found using what precedes seems more difficult. The following code has been tested with pdfTeX, XeTeX and LuaTeX.

The difficulty is that \str_set_convert:Nnnn \whatever { ⟨comma-list of code points⟩ } { clist } { default } stores non-space character tokens with catcode 12 in \whatever, which normally can't be used to typeset non-ASCII characters with 8-bit engines. For pdfTeX, we need active characters for the non-ASCII chars (or a suitable mapping from code points to \textcommandsomething, which I don't have at hand).

Note: this will not work for chars in the ASCII range (0-127) where the font encoding differs from ASCII (this is the case at least for |, <, >, and " in the OT1 encoding; T1 is nicer in this respect).

\documentclass{article}
\usepackage{xcolor}
% Font setup: only needed for typesetting the sample string
\usepackage{iftex}
\ifpdftex
  \usepackage{lmodern}
  \usepackage[T1]{fontenc}
\fi

\ExplSyntaxOn
\cs_new_protected:Npn \jlb_iterate_over_code_points:nnn #1#2#3
  {
    \str_set_convert:Nnnn \l_tmpa_clist {#2} {#1} { clist }
    \clist_map_inline:Nn \l_tmpa_clist {#3}
  }

\tl_new:N \l__jlb_prepare_for_typesetting_tl
\cs_generate_variant:Nn \tl_analysis_map_inline:nn { V }

% Store in tl var #1 whatever is needed to typeset the Unicode code point #2
% (given as an integer denotation—e.g., a decimal representation)
\cs_new_protected:Npn \jlb_prepare_for_typesetting:Nn #1#2
  {
    \str_set_convert:Nnnn \l__jlb_prepare_for_typesetting_tl {#2} { clist }
      { default }

    \bool_lazy_and:nnTF { \int_compare_p:nNn {#2} > { 127 } }
                        { \sys_if_engine_pdftex_p: }
      {
        \tl_clear:N #1
        \tl_analysis_map_inline:Vn \l__jlb_prepare_for_typesetting_tl
           {
             % \char_generate:nn does its job in two expansion steps
             \exp_args:NNNo \exp_args:NNo
             \tl_put_right:Nn #1 { \char_generate:nn {##2} { 13 } }
           }
      }
      {
        % Char token (32, 12) isn't a space token → let's special-case spaces
        \int_compare:nNnTF {#2} = { 32 }
          { \tl_set:Nn #1 { \scan_stop: \c_space_token } }
          { \tl_set_eq:NN #1 \l__jlb_prepare_for_typesetting_tl }
      }
  }

\tl_new:N \l__jlb_alternate_one_char_tl
\int_new:N \l__jlb_alternate_counter_int

\NewDocumentCommand \alternate { m m m }
  {
    \int_zero:N \l__jlb_alternate_counter_int

    \jlb_iterate_over_code_points:nnn { default } {#3}
      {
        % Make \__jlb_alternate_mapping_func:n alternate between #1 and #2
        \int_if_even:nTF { \l__jlb_alternate_counter_int }
          { \cs_set_eq:NN \__jlb_alternate_mapping_func:n #1 }
          { \cs_set_eq:NN \__jlb_alternate_mapping_func:n #2 }

        % Store suitable tokens in \l__jlb_alternate_one_char_tl to allow
        % typesetting the character whose Unicode code point is ##1.
        \jlb_prepare_for_typesetting:Nn \l__jlb_alternate_one_char_tl {##1}
        % Use the result with \__jlb_alternate_mapping_func:n
        \__jlb_alternate_mapping_func:n { \l__jlb_alternate_one_char_tl }

        \int_incr:N \l__jlb_alternate_counter_int
      }
  }
\ExplSyntaxOff

\newcommand*{\typesetInBlue}[1]{\textcolor{blue}{#1}}
\newcommand*{\typesetInRed}[1]{\textcolor{red}{#1}}

\begin{document}

\alternate{\typesetInBlue}{\typesetInRed}{aàáâãäåāăąæ}

\end{document}

enter image description here

10
  • Nice use of \str_set_convert:Nnn, by the way :)
    – Joseph Wright
    Jun 6, 2022 at 9:02
  • For instance, an é can be written using two Unicode code points … Isn't that where Unicode Normalization comes into play … ? unicode.org/reports/tr15
    – Ingmar
    Jun 6, 2022 at 9:10
  • @Ingmar Yes: I'm not an Unicode expert, but my understanding is that in a given normalized form (there are several kinds), only one representation is allowed for a given “string”.
    – frougon
    Jun 6, 2022 at 9:14
  • 2
    I added an example with combining characters. Fun fact: here are two characters in four code points (using combining accents in both cases): én̈. In Emacs, using the left and right arrow keys twice is enough to get over them. However, if I place the cursor (called “the point” in Emacs) after these characters, then press Backspace three times, I successively see én̈, én, , and finally e. This shows that, in Emacs, Backspace erases combining code points separately from the code point they apply to.
    – frougon
    Jun 6, 2022 at 9:51
  • 2
    @frougon We do something a bit similar to create the case changing code in loading expl3: take a look in l3text to see how we do it. BTW, you can use \sys_if_engine_pdftex_p: rather than the legacy if.
    – Joseph Wright
    Jun 6, 2022 at 14:15
7

When dealing with -- as you put it -- "exotic" (but still single-glyph and utf8-encoded, right?) characters at a low level, it's best not to work with pdf(La)TeX; instead, do work with a TeX engine that's natively utf8-aware. I know of two such engines: XeTeX and LuaTeX.

If you choose to work with LuaTeX, you can even use Lua's library of powerful string functions, as needed, to extend the macros provided by the xstring package.

Aside: This approach -- i.e., using the macros of the xstring package, including \StrLen, under LualaTeX or XeLaTeX -- assumes that the visible "characters" are encoded as "single" glyphs. Unicode and UTF8 also have something called "combining" characters, such as . When applied to a "combining character", the output of \StrLen will be 2 -- or even more, in case of multiply-combining characters.

enter image description here

% !TEX TS-program = lualatex
\documentclass{article}
\usepackage{xstring} % for '\StrLen' macro
% create an alternative to '\StrLen':
\newcommand*\StrLenAlt[1]{\directlua{tex.sprint(unicode.utf8.len("#1"))}}

\begin{document}
---\StrLen{aàáâãäåāăąæ}---\StrLenAlt{aàáâãäåāăąæ}---
\end{document}
6
  • It's so simple in lua! I was just wondering why not to completely avoid xstring in your case and just define our own \StrLen? This would definitely save some compiling time I guess :)
    – Niranjan
    Jun 6, 2022 at 11:55
  • Also unless someone needs to use a blank line (or a \par token) in the argument of this command, wouldn't \newcommand* be a better choice?
    – Niranjan
    Jun 6, 2022 at 12:00
  • @Niranjan - On your first question: My answer was indeed meant to demonstrate that a huge plus of Lua(La)TeX is that it gives immediate access to a very well documented and widely used library of string functions. On the second question: I'm afraid I don't understand why it would be a "better choice" to exclude a \par token from the argument of \StrLenAlt. What is the advantage, in your view, of the "better choice"?
    – Mico
    Jun 6, 2022 at 13:04
  • 1
    and regarding the second one the advantage that I see is the users will be prompted with an error if they mistakenly forget to provide any argument and follow the command with a new line, e.g., The word programming has the following number of characters: \StrLenAlt followed by a blank line. Here the user forgot to add programming as an argument. Still it prints 0. A new user not having enough knowledge would be very surprised and in a long code it would probably just be ignored without any attention. Whereas if you use \newcommand* the user will be prompted with an error.
    – Niranjan
    Jun 6, 2022 at 14:54
  • 1
    @Niranjan - Many thanks for these clarifications. I've gone ahead and changed the directive that creates \StrLenAlt from \newcommand to \newcommand*.
    – Mico
    Jun 6, 2022 at 18:08
6

As other answers wrote, counting characters in Unicode is somewhat ill-defined. While other answers showed how to count Unicode codepoints, this can lead to rather unexpected results. E.g. ä might be one or two codepoints depending on used whether its the combined character or an a with a combining diaeresis. While this can sometimes be solved by normalizing all documents to NFC which avoids such combining character, this fails when characters are used for which no precombined forms exists.

For pdfTeX this doesn't really matter since combining characters aren't really supported anyway, but in LuaTeX this should be supported.

A often more user friendly way to count characters in Unicode is to count Unicode "grapheme clusters". These are clusters of Unicode codepoints which roughly correspond to what's commonly considered to be a character. My lua-uni-algos package contains Lua utilities to find grapheme cluster boundaries in some text, but doesn't provide a TeX based interface.

But it can be used to write such an interface (obviously requires LuaTeX):

\documentclass{article}
\usepackage{iftex}
\RequireLuaTeX
\directlua{
  % You can ignore this block, it's just a technical helper to allow Lua based "macro expansion"
  local match = 0x1A00000
  local end_match = 0x1C00000
  local arg1_tok = token.new(1, token.command_id'car_ret')
  local function insert_arg(toks, arg1)
    if not toks[1] == match or not toks[2] == end_match then
      error"Misuse"
    end
    local result = {}
    local j = 1
    local arg_length = \csstring\#arg1
    for i = 3, \csstring\#toks do
      local t = toks[i]
      if t == arg1_tok then
        table.move(arg1, 1, arg_length, j, result)
        j = j + arg_length
      else
        result[j] = t
        j = j + 1
      end
    end
    return result
  end

  % Load the grapheme handler from lua-uni-algos
  local read_codepoint = require'lua-uni-graphemes'.read_codepoint
  % Which category codes should be allowed? We treat everything except
  % spaces, letters and "other" chars as an error.
  local letter_like = {
    [token.command_id'spacer'] = true,
    [token.command_id'letter'] = true,
    [token.command_id'other_char'] = true,
  }

  % Some boilerplate to define \MapUnicodeGraphemes as a command from Lua
  local func = luatexbase.new_luafunction'MapUnicodeGraphemes'
  token.set_lua('MapUnicodeGraphemes', func)
  lua.get_functions_table()[func] = function()
    % Here we really start
    % ====================
    % First read the text we want to map
    local text = token.scan_toks(false, true)
    % Then scan what each grapheme cluster should be mapped to.
    % Since this is scanned as a macro (like in \def), we have to insert #1 first to
    % indicate that #1 is allowed inside.
    token.put_next(token.create(\number`\#), token.create(\number`\1))
    local mapping = token.scan_toks(true, false)
    % Some variables:
    % - `state` is a blackbox needed by lua-uni-graphemes
    % - `new_cluster` will indicate if the current codepoint starts a new cluster
    % - `start_of_cluster` is the index where the last cluster started
    local state, new_cluster, start_of_cluster
    % Iter
    for i, tok in ipairs(text) do
      if letter_like[tok.command] then
        % If this is actually a letter, pass it to lua_uni_graphemes
        new_cluster, state = read_codepoint(tok.index, state)
        if new_cluster then
          % We have a new cluster. If we had a previous cluster, then that cluster finished with the previous character. Map it according to mapping and send it to TeX.
          if start_of_cluster then
            tex.sprint(insert_arg(mapping, table.move(text, start_of_cluster, i - 1, 1, {})))
          end
          % Then record the beginning of this cluster
          start_of_cluster = i
        end
      else
        % If some other token appears raise and error
        tex.error('Ignoring unexpected token in argument')
      end
    end
    % Finally make sure that the final cluster also gets mapped and send to TeX
    if start_of_cluster then
      tex.sprint(insert_arg(mapping, table.move(text, start_of_cluster, \csstring\#text, 1, {})))
    end
  end
}

\NewExpandableDocumentCommand \countGraphemes { m }
  {%
    % With the \MapUnicodeGraphemes helper we can map every grapheme cluster to +1, such that we can form an expression indicating the number of clusters
    \number\numexpr0\MapUnicodeGraphemes {#1}{+1}\relax
  }

\begin{document}

There are \countGraphemes{aàáâãäåāăąæa̧̯̅̈̏} grapheme clusters in “aàáâãäåāăąæa̧̯̅̈̏”. These are:
\MapUnicodeGraphemes{aàáâãäåāăąæa̧̯̅̈̏}{#1, } and that's it.

\end{document}

enter image description here

2
  • Wow! Tiny thing: either add a % after \NewExpandableDocumentCommand \countGraphemes { m } { or use \ExplSyntaxOn...
    – frougon
    Jun 6, 2022 at 22:45
  • 1
    @frougon Thanks, fixed. Jun 7, 2022 at 2:44
4

This is not a general issue with accented ("exotic") characters, only with some characters, notably ā and other "overlined" characters such as ē, ī, etc, and "caroned" characters such as ǎ. Notice:

\documentclass{article}

\usepackage{xstring}
\usepackage[T1]{fontenc}

\begin{document}
\StrLen{aàáâãäåăąæ}
\end{document}

correctly returns 10, whereas \StrLen{ā} returns 47.

1
  • Good obervation, +1 as I've incorporated this at the start of my answer. :-)
    – frougon
    Jun 6, 2022 at 14:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .