Testing if a string is a hexadecimal string in LaTeX3: code review, optimization, expandability, and protection

Question

Consider the code in LaTeX3. It tests whether a string is a hexadecimal string (optionally also testing the length of that string) (a string where every character is either 0-9 or A-F).

\documentclass[10pt]{article}
\usepackage{xparse}

\ExplSyntaxOn
\NewDocumentCommand{\ishexadecimal}{o m m m}{
    \is_hexadecimal_wrapper:eeTF{#1}{\tl_trim_spaces:e{#2}}{#3}{#4}
}
\str_const:Nn \hex_character_set{0123456789ABCDEF}
\cs_new:Nn \is_hexadecimal_wrapper:nnTF{
    \is_hexadecimal:eeTF{#1}{\str_uppercase:n{#2}}{#3}{#4}
}
\cs_new:Nn \is_hexadecimal:nnTF{
    \bool_lazy_or:nnTF{\tl_if_novalue_p:n{#1}}{
        \int_compare_p:nNn{\str_count:n{#2}}={#1}
    }{
        \bool_set_true:N \l_tmpa_bool
        \str_map_function:nN{#2}{\is_hexadecimal_loop:n}
        \bool_if:NTF \l_tmpa_bool{#3}{#4}
    }{#4}
}
\cs_new:Nn \is_hexadecimal_loop:n{
    \str_if_in:NeF \hex_character_set{#1}{
        \bool_set_false:N \l_tmpa_bool
        \str_map_break:
    }
}
\cs_generate_variant:Nn \str_if_in:NnF{NeF}
\cs_generate_variant:Nn \is_hexadecimal:nnTF{eeTF}
\cs_generate_variant:Nn \is_hexadecimal_wrapper:nnTF{eeTF}
\ExplSyntaxOff

\begin{document}
\ishexadecimal{00AAFF98}{TRUE}{FALSE}% Test if the string is a hexadecimal string: TRUE
\ishexadecimal[6]{00AAFF98}{TRUE}{FALSE}% Test if the string is a hexadecimal string of length 6: FALSE
\ishexadecimal[8]{00AAFF98}{TRUE}{FALSE}% Test if the string is a hexadecimal string of length 8: TRUE
\end{document}

I have several questions on how to make that code better, especially to better understand LaTeX3, protection, and expandability:

QUESTION 1: Are there easy and obvious optimizations (including unnecessary step) that can be done to help compilation time (if this function is low level and called thousands of time in a document)?

QUESTION 2: Should any of the defined function be protected? If so, why? If so, what would the final code look like?

QUESTION 3: When I call \token_if_expandable:NTF on any of the defined functions in that document, it always return true. Is that the case? Are all the functions fully expandable?

QUESTION 4: If all functions are expandable, should I make \ishexadecimal a \NewExpandableDocumentCommand? Is so why, if not, why?

QUESTION 5: Should I use \tl_trim_spaces:e or >{\TrimSpaces}? What would be the best practice and why?

An improved version of the code applying good practices (with explanation) would be very nice to understand how to better code in LaTeX3.

EDIT:

QUESTION 6: If the only reason why this function is not expandable is the need for assignment in the context of \str_map_function (\bool_set_true:N \l_tmpa_bool) is there an usual way to rewrite this kind of pattern to avoid the use of assignment?

Answer 1

You want to define conditionals.

\documentclass[10pt]{article}

\ExplSyntaxOn
\NewExpandableDocumentCommand{\ishexadecimal}{o m m m}
 {
  \IfNoValueTF {#1}
   {% no check on lenght
    \vincent_ishex:eTF { \tl_trim_spaces:e {#2} } {#3} {#4}
   }
   {% check the length
    \vincent_ishex_length:eeTF {#1} {#2}
     {% length matches
      \vincent_ishex:eTF { \tl_trim_spaces:e {#2} } {#3} {#4}
     }
     {% length doesn't match
      #4
     }
   }
 }

\cs_generate_variant:Nn \str_map_function:nN {e}

\group_begin:
\char_set_catcode_other:n { `\^^A }

\prg_new_conditional:Nnn \vincent_ishex:n { p,T,F,TF }
 {
  \str_map_function:eN { \str_uppercase:n {#1} ^^A } \__vincent_ishex_digit:N
 }
\prg_generate_conditional_variant:Nnn \vincent_ishex:n {e} { p,T,F,TF }

\cs_new:Nn \__vincent_ishex_digit:N
 {
  \str_case:nnF {#1}
   {
    {0}{}{1}{}{2}{}{3}{}{4}{}{5}{}{6}{}{7}{}{8}{}{9}{}{A}{}{B}{}{C}{}{D}{}{E}{}{F}{}
    {^^A}{\prg_return_true:}
   }
   {
    \str_map_break:n { \prg_return_false: }
   }
 }
\group_end:

\prg_new_conditional:Nnn \vincent_ishex_length:nn { p,T,F,TF }
 {
  \int_compare:nNnTF {#1} = { \str_count:n {#2} }
   {
    \prg_return_true:
   }
   {
    \prg_return_false:
   }
 }
\prg_generate_conditional_variant:Nnn \vincent_ishex_length:nn {ee} { p,T,F,TF }
\ExplSyntaxOff

\newcommand{\myhex}{123AC}
\newcommand{\mylenA}{4}
\newcommand{\mylenB}{5}

\begin{document}

\ishexadecimal{00AAFF98}{TRUE}{FALSE} (true)

\ishexadecimal[6]{00AAFF98}{TRUE}{FALSE} (false)

\ishexadecimal[8]{00AAFF98}{TRUE}{FALSE} (true)

\ishexadecimal{pqr}{TRUE}{FALSE} (false)

\ishexadecimal{\myhex}{TRUE}{FALSE} (true)

\ishexadecimal[\mylenA]{\myhex}{TRUE}{FALSE} (false)

\ishexadecimal[\mylenB]{\myhex}{TRUE}{FALSE} (true)

\edef\test{\ishexadecimal[\mylenB]{\myhex}{TRUE}{FALSE}}
\texttt{\meaning\test}

\end{document}

The idea is to uppercase the string; then we call \str_map_function:nN to the string augmented by ^^A, which cannot be directly typed in a normal document; hexadecimal digits make the loop to go on, whereas ^^A issues \prg_return_true: because we've reached the end of the string; otherwise the loop is stopped and \prg_return_false: is issued.

The check on the length can be done independently when the optional argument is present.

I think that the code above answers all your subquestions.

Answer 2

1. See below.

2. Yes, there are functions that should be defined protected in your question, for instance \cs_new:Nn \is_hexadecimal:nnTF (and as a result all functions calling it). The reason that's the case is rather simple: It needs an unexpandable function (\bool_set_true:N) itself. The rules whether something should be defined protected boil down to the following:

   1. If a function uses an unexpandable function it should be defined protected.
   2. If a function needs an assignment it should be defined protected (because those are not expandable -- see rule 1)

   So inside expl3 you can check for every function whether it's marked as expandable in interface3.pdf, if it isn't and your function uses it yours should be protected as well.

   However, there are exemptions from these rules, those are very few and it boils down to: If you need the primitives \noalign or \omit inside your function you must not be protected until you hit those. If you do some really low level optimisations there might be few further cases in which you don't want to be protected even though you're using assignments (though I don't recall any such case I encountered myself I don't want to rule it out from the get-go).

3. \token_if_expandable:NTF is a really simple test internally, you can't use it to verify whether a function you defined should be protected or not. In short it reports all macros as expandable (even if they are protected).

4. Not all of your functions are expandable (see 2.) so this question doesn't apply and you should continue to use \NewDocumentCommand.

5. This depends on what you want to do. If your low level function should ignore leading and trailing spaces use \tl_trim_spaces:n in it. If it shouldn't but your document level command should use >{\TrimSpaces}. Note that if you use an argument processor you can't use \NewExpandableDocumentCommand.

6. Not only \bool_set_(true|false):N are problematic, your usage of \str_if_in:NnTF also isn't expandable. But yes, you could rewrite that bool-setting portion of your code, for instance you could use \bool_if:nTF { \str_map_function:nN {#1} \your_foo:n } { true } { false } and have your \your_foo:n do a \str_map_break:n { \c_(true|false)_bool } to end the loop once it knows the result. For that its check would need to be expandable itself as well though, which it isn't because of \str_if_in:NnTF.

---

This is an optimised variant checking whether some input would form a valid hexadecimal number. This here is (as of writing this) roughly 60 times faster than the code provided in your question, and roughly 50 times faster than the code provided by @egreg, all while being fully expandable (just like @egreg's).

However the code here uses a hand created loop, a low level if_hexdigit function (which might break if #1 is a macro, but is only applied on stringified contents here, for which it's fine), a fast check for an existing hash entry (more or less a shortcut version of \cs_if_exist:cTF which doesn't care whether the current meaning is \scan_stop:) and otherwise a few low level optimisations. Note that \prg_new_conditional:Npnn uses a speed optimised version of the branching code if your replacement text ends in \prg_return_true: \else: \prg_return_false: \fi: which is the case for every conditional defined in this code.

There is a small difference in behaviour between my document level definition and yours, namely the input isn't fully expanded. If you want this use \tl_trim_spaces_apply:eN instead of \tl_trim_spaces_apply:nN.

\documentclass{article}

\ExplSyntaxOn
\NewExpandableDocumentCommand \ishexstringTF { o m m m }
  {
    % \tl_tirm_spaces_apply:nN is faster than e-expanding \tl_trim_spaces:n
    \IfNoValueTF {#1}
      { \tl_trim_spaces_apply:nN {#2} \vincent_if_hexstring:nTF }
      { \tl_trim_spaces_apply:nN {#2} \vincent_if_hexstring:nnTF {#1} }
      {#3} {#4}
  }

% check for a string consisting only of hex digits and having a defined length
\prg_new_conditional:Npnn \vincent_if_hexstring:nn #1#2 { TF, T, F, p }
  {
    \if:w
        ?
        % we can ignore spaces for the count, as everything containing spaces is
        % false in \vincent_if_hexstring:nF.
        \int_compare:nNnT {#2} = { \str_count_ignore_spaces:n {#1} }
          { \vincent_if_hexstring:nF {#1} }
          !
        ?
      \prg_return_true:
    \else:
      \prg_return_false:
    \fi:
  }

\prg_new_conditional:Npnn \vincent_if_hexstring:n #1 { TF, T, F, p }
  {
    \if:w
        ? \exp_after:wN \__vincent_if_hexstring:w \tl_to_str:n {#1} ~ \q_stop ?
      \prg_return_true:
    \else:
      \prg_return_false:
    \fi:
  }
% checks whether no spaces are contained, if true starts the loop else return
% false (!)
\cs_new:Npn \__vincent_if_hexstring:w #1 ~ #2 \q_stop
  {
    \tl_if_empty:nF {#2} !
    \__vincent_if_hexstring_loop:N #1 \q_stop
  }
\cs_new:Npn \__vincent_if_hexstring_loop:N #1
  {
    % fast check for end of loop
    \use_none_delimit_by_q_stop:w #1 \use_none:nnnnn \q_stop
    \vincent_if_hexdigit:NF {#1} { \__vincent_if_hexstring_char:N #1 }
    \__vincent_if_hexstring_loop:N
  }
\cs_new:Npn \__vincent_if_hexstring_char:N #1
  { \vincent_if_hashentry:nF { __vincent_hash_char_ #1 : } ! }
% we only need special casing for lower case hex digits as TeX doesn't recognise
% those in "<num> notation.
\tl_map_inline:nn { abcdef }
  { \cs_new_eq:cN { __vincent_hash_char_ #1 : } \scan_stop: }

% check whether TeX thinks #1 is a hex digit (that only includes upper case A-F)
\prg_new_conditional:Npnn \vincent_if_hexdigit:N #1 { TF, T, F, p }
  {
    \if_int_compare:w 1 < "1 #1 \exp_stop_f:
      \prg_return_true:
    \else:
      \prg_return_false:
    \fi:
  }
% check whether a given hash table entry exists, regardless of its definition
\prg_new_conditional:Npnn \vincent_if_hashentry:n #1 { TF, T, F, p }
  {
    \if_cs_exist:w #1 \cs_end:
      \prg_return_true:
    \else:
      \prg_return_false:
    \fi:
  }
\ExplSyntaxOff

\begin{document}
\ishexstringTF{00AAFF98}{TRUE}{FALSE} % Test if the string is a hexadecimal string: TRUE
\ishexstringTF[6]{00AAFF98}{TRUE}{FALSE} % Test if the string is a hexadecimal string of length 6: FALSE
\ishexstringTF[8]{00AAFF98}{TRUE}{FALSE} % Test if the string is a hexadecimal string of length 8: TRUE
\end{document}

Answer 3

You asked about optimization of compilation time. So, you can try to use TeX primitive features directly, no via expl3 language. Expl3 is a new language inside TeX language then the result will never be ideally optimized.

You can try this:

% usage: \ishexstring {string} {true} {false}
\def\ishexstring#1{\ifx^#1^\iftrue \ishexF\fi\fi \ishexstringA#1\end}
\def\ishexstringA#1{\ifx#1\end \ishexT
   \else \ifnum 5<"1#1 \expandafter\expandafter\expandafter \ishexstringA
         \else \ishexF 
   \fi \fi
}
\def\ishexT #1\fi\fi#2#3{\fi #2}
\def\ishexF #1\end #2#3{\fi\fi #3}

% \strlen {string} expands to the length of the string
\def\strlen#1{\strlenA 0#1\end}
\def\strlenA#1#2{\ifx\end#2#1\else 
   \afterfi{\expandafter\strlenA\expandafter{\the\numexpr#1+1}}\fi
}
\def\afterfi#1#2\fi{\fi #1}

\def\ishexstringL[#1]#2#3#4{\ifnum\strlen{#2}=#1 \ishexstring{#2}{#3}{#4}\else #4\fi}

\ishexstring{00AAFF98}{TRUE}{FALSE} (true)

\ishexstring{00AAFG98}{TRUE}{FALSE} (false)

\ishexstringL[6]{00AAFF98}{TRUE}{FALSE} (false)

\ishexstringL[8]{00AAFF98}{TRUE}{FALSE} (true)

\bye

The TeX primitive features are used here directly and macros are expandable. You can try to run \tracingall like this:

{\tracingall \ishexstring{00AAFF98}{TRUE}{FALSE} (true) }

and look to the log file. There are 128 lines with traced macros. If you do the same with Skilmon's solution (started from \ishexstringTF expansion), you see 588 lines and if you do this with egreg's solution you see 7859 lines in the log file.