Alternative to detokenize working with special unicode characters

Question

I have defined new command using \NewDocumentCommand which represents description of node in graph. This contains name (provided as argument), some text description (provided as argument) and links to other nodes (specified as semicolon separated list of names in argument). From answer to this question, I have discovered \detokenize command, which allows me to properly display node names containing underscores. (Also I use this in \paragraph names, \label names and \references.)

Problem is, when I use \detokenize on text description, some Unicode characters are replaced with wrong characters. (For example, softvér gets replaced by softvŠr.) Is there some alternative to \detokenize? I only need to automatically escape < and > that are present in some descriptions. I could use \textless and textgreater, but I had to use detokenize for names (so that referencing would work, I could not escape it properly as it is argument of macro), so I would like to have some consistency and provide description un-escaped.

edit here is small example:

\documentclass[12pt, oneside]{book}
\usepackage[a4paper,top=2.5cm,bottom=2.5cm,left=3.5cm,right=2cm]{geometry}
\usepackage[utf8]{inputenc}
\usepackage[IL2]{fontenc}
\usepackage[hidelinks,breaklinks]{hyperref}
\usepackage[slovak]{babel}
\usepackage{nameref}
\usepackage{xparse}
\usepackage{ifthen}
\linespread{1.25}

\NewDocumentCommand\mynode{mm>{\SplitList{;}}m>{\SplitList{;}}m}
{
    \paragraph[\detokenize{#1}]{\detokenize{#2} (\texttt{\detokenize{#1}}):}
    \label{\detokenize{#1}}
    \begin{itemize}
        \item vstupné hrany:
        \ProcessList{#3}{ \insertdep }
        \item výstupné hrany:
        \ProcessList{#4}{ \insertdep }
    \end{itemize}
}
\newcommand{\insertdep}[1]{
\ifthenelse{\equal{}{#1}}{\texttt{-}}{\texttt{\nameref{\detokenize{#1}}};}
}

\begin{document}
\mynode{root}{Koreň}{root}{gui}
\mynode{gui}{Grafické rozhranie}{root}{nice_gui;fancy_gui}
\mynode{nice_gui}{Pekné grafické rozhranie}{gui}{}
\mynode{fancy_gui}{This text contains less and greater characters:<>}{gui}{}
\end{document}

\detokenize{#2} breaks special Slovak characters (compared to using only #2). I have noticed, that it does not even fix problems with _<>. I have noticed, that when I use \verb|#2| instead of \detokenize{#2}, Slovak characters and <> work, but it obviously changes formatting and for some reason adds | at the end of text, and underscores still make latex to insert math mode and make subscript.

I'm using pdflatex to compile.

Answer 1

It's a bit more complicated than using \detokenize, particularly with IL2. By the way, do you really need it?

Here's a way that preprocesses the input so to change _<> into proper commands.

\documentclass[12pt, oneside]{book}
\usepackage[a4paper,top=2.5cm,bottom=2.5cm,left=3.5cm,right=2cm]{geometry}
\usepackage[utf8]{inputenc}
\usepackage[IL2]{fontenc}
\usepackage[slovak]{babel}
\usepackage{nameref}
\usepackage{xparse}
\usepackage[hidelinks,breaklinks]{hyperref}

\ExplSyntaxOn

\NewDocumentCommand\mynode{mmmm}
 {
  \mynode_main:nnnn { #1 } { #2 } { #3 } { #4 }
 }

\seq_new:N \l_mynode_input_seq
\seq_new:N \l_mynode_output_seq
\tl_new:N \l_mynode_node_tl
\tl_new:N \l_mynode_text_tl

\cs_new_protected:Nn \mynode_main:nnnn
 {
  \seq_set_split:Nnn \l_mynode_input_seq { ; } { #3 }
  \seq_set_split:Nnn \l_mynode_output_seq { ; } { #4 }
  \tl_set:Nn \l_mynode_node_tl { #1 }
  \mynode_replace:N \l_mynode_node_tl
  \tl_set:Nn \l_mynode_text_tl { #2 }
  \mynode_replace:N \l_mynode_text_tl

  \mynode_paragraph:VV \l_mynode_node_tl \l_mynode_text_tl
  \label{ \tl_to_str:n { #1 } }

  \begin{itemize}
    \item vstupné~hrany:~
    \seq_map_variable:NNn \l_mynode_input_seq \l_mynode_node_tl
     {
      \mynode_print:N \l_mynode_node_tl
     }
    \item výstupné~hrany:~
    \seq_map_variable:NNn \l_mynode_output_seq \l_mynode_node_tl
     {
      \mynode_print:N \l_mynode_node_tl
     }
  \end{itemize}
 }

\cs_new_protected:Nn \mynode_paragraph:nn
 {
  \paragraph[#1]{#2~(\texttt{#1}):}
 }
\cs_generate_variant:Nn \mynode_paragraph:nn { VV }

\cs_new_protected:Nn \mynode_print:N
 {
  \texttt
   {
    \tl_if_empty:NTF #1
      { - }
      { \nameref { \tl_to_str:N #1 } }
   };~
 }

\cs_new_protected:Nn \mynode_replace:N
 {
  \regex_replace_all:nnN { \_ } { \c{_} } #1
  \regex_replace_all:nnN { \< } { \c{textless} } #1
  \regex_replace_all:nnN { \> } { \c{textgreater} } #1
 }

\NewDocumentCommand{\change}{m}
 {
  \tl_set:Nn \l_mynode_text_tl { #1 }
  \mynode_replace:N \l_mynode_text_tl
  \tl_use:N \l_mynode_text_tl
  %\tl_analysis_show:N #1 %%% for debugging
 }
\ExplSyntaxOff

\begin{document}

\mynode{root}{Koreň}{root}{gui}

\mynode{gui}{Grafické rozhranie}{root}{nice_gui;fancy_gui}

\mynode{nice_gui}{Pekné grafické rozhranie}{gui}{}

\mynode{fancy_gui}{This text contains less and greater characters:<>}{gui}{}

\mynode{<test>}{Some text with \texttt{under_score}}{<test>}{}
\end{document}

Answer 2

Actually, you have two separate problems: one with font encoding and one with text encoding. Unfortunately, I can help you only with the second one.

\detokenize is not UTF8-friendly, but sometimes you can replace it with the following command:

\newcommand{\utffriendlydetokenize}[1]{%
\scantokens{%
\catcode`\_=12%
\catcode`\^=12%
\catcode`\{=12%
\catcode`\}=12%
\catcode`\&=12%
\catcode`\$=12%
\catcode`\#=12%
\catcode`\~=12%
\catcode`\\=12%
#1%
}%
}

(source: https://gitlab.com/Nickkolok/biblatex2bibitem/-/blob/master/biblatex2bibitem.sty)

In your case the only row that should be changed is:

\paragraph[\detokenize{#1}]{\utffriendlydetokenize{#2} (\texttt{\detokenize{#1}}):}

And you get your ň back:

However, it does not fix the problem with <>_, because the problem is caused by the codes of the symbols in specified fontenc. (See also Strange behaviour with \detokenize and underscore) The only additional solution that can guess about is to redefine _<> to be active characters by playing with their catcodes, to map them to the corresponding commands and to correct the definition \utffriendlydetokenize for the chosen catcodes.