# Alternative to detokenize working with special unicode characters

I have defined new command using \NewDocumentCommand which represents description of node in graph. This contains name (provided as argument), some text description (provided as argument) and links to other nodes (specified as semicolon separated list of names in argument). From answer to this question, I have discovered \detokenize command, which allows me to properly display node names containing underscores. (Also I use this in \paragraph names, \label names and \references.)

Problem is, when I use \detokenize on text description, some Unicode characters are replaced with wrong characters. (For example, softvér gets replaced by softvŠr.) Is there some alternative to \detokenize? I only need to automatically escape < and > that are present in some descriptions. I could use \textless and textgreater, but I had to use detokenize for names (so that referencing would work, I could not escape it properly as it is argument of macro), so I would like to have some consistency and provide description un-escaped.

edit here is small example:

\documentclass[12pt, oneside]{book}
\usepackage[a4paper,top=2.5cm,bottom=2.5cm,left=3.5cm,right=2cm]{geometry}
\usepackage[utf8]{inputenc}
\usepackage[IL2]{fontenc}
\usepackage[slovak]{babel}
\usepackage{nameref}
\usepackage{xparse}
\usepackage{ifthen}

\NewDocumentCommand\mynode{mm>{\SplitList{;}}m>{\SplitList{;}}m}
{
\paragraph[\detokenize{#1}]{\detokenize{#2} (\texttt{\detokenize{#1}}):}
\label{\detokenize{#1}}
\begin{itemize}
\item vstupné hrany:
\ProcessList{#3}{ \insertdep }
\item výstupné hrany:
\ProcessList{#4}{ \insertdep }
\end{itemize}
}
\newcommand{\insertdep}[1]{
\ifthenelse{\equal{}{#1}}{\texttt{-}}{\texttt{\nameref{\detokenize{#1}}};}
}

\begin{document}
\mynode{root}{Koreň}{root}{gui}
\mynode{gui}{Grafické rozhranie}{root}{nice_gui;fancy_gui}
\mynode{nice_gui}{Pekné grafické rozhranie}{gui}{}
\mynode{fancy_gui}{This text contains less and greater characters:<>}{gui}{}
\end{document}


\detokenize{#2} breaks special Slovak characters (compared to using only #2). I have noticed, that it does not even fix problems with _<>. I have noticed, that when I use \verb|#2| instead of \detokenize{#2}, Slovak characters and <> work, but it obviously changes formatting and for some reason adds | at the end of text, and underscores still make latex to insert math mode and make subscript.

I'm using pdflatex to compile.

• your question is not that clear, you can use _, < and > in label names without needing to do anything, \label{foo_bar}... \ref{foo_bar} works. – David Carlisle Apr 21 '19 at 12:41
• Show a complete minimal example, and also say how you are compiling (pdftex or luatex) – Ulrike Fischer Apr 21 '19 at 12:41
• I have added small working example at the end of post. I have tried to remove any unnecessary usepackage commands. Sorry if some are left. – user224348 Apr 21 '19 at 13:49
• as you have seen, \detokenize completely disables latex's encoding support. – David Carlisle Apr 21 '19 at 14:53

It's a bit more complicated than using \detokenize, particularly with IL2. By the way, do you really need it?

Here's a way that preprocesses the input so to change _<> into proper commands.

\documentclass[12pt, oneside]{book}
\usepackage[a4paper,top=2.5cm,bottom=2.5cm,left=3.5cm,right=2cm]{geometry}
\usepackage[utf8]{inputenc}
\usepackage[IL2]{fontenc}
\usepackage[slovak]{babel}
\usepackage{nameref}
\usepackage{xparse}

\ExplSyntaxOn

\NewDocumentCommand\mynode{mmmm}
{
\mynode_main:nnnn { #1 } { #2 } { #3 } { #4 }
}

\seq_new:N \l_mynode_input_seq
\seq_new:N \l_mynode_output_seq
\tl_new:N \l_mynode_node_tl
\tl_new:N \l_mynode_text_tl

\cs_new_protected:Nn \mynode_main:nnnn
{
\seq_set_split:Nnn \l_mynode_input_seq { ; } { #3 }
\seq_set_split:Nnn \l_mynode_output_seq { ; } { #4 }
\tl_set:Nn \l_mynode_node_tl { #1 }
\mynode_replace:N \l_mynode_node_tl
\tl_set:Nn \l_mynode_text_tl { #2 }
\mynode_replace:N \l_mynode_text_tl

\mynode_paragraph:VV \l_mynode_node_tl \l_mynode_text_tl
\label{ \tl_to_str:n { #1 } }

\begin{itemize}
\item vstupné~hrany:~
\seq_map_variable:NNn \l_mynode_input_seq \l_mynode_node_tl
{
\mynode_print:N \l_mynode_node_tl
}
\item výstupné~hrany:~
\seq_map_variable:NNn \l_mynode_output_seq \l_mynode_node_tl
{
\mynode_print:N \l_mynode_node_tl
}
\end{itemize}
}

\cs_new_protected:Nn \mynode_paragraph:nn
{
\paragraph[#1]{#2~(\texttt{#1}):}
}
\cs_generate_variant:Nn \mynode_paragraph:nn { VV }

\cs_new_protected:Nn \mynode_print:N
{
\texttt
{
\tl_if_empty:NTF #1
{ - }
{ \nameref { \tl_to_str:N #1 } }
};~
}

\cs_new_protected:Nn \mynode_replace:N
{
\regex_replace_all:nnN { \_ } { \c{_} } #1
\regex_replace_all:nnN { \< } { \c{textless} } #1
\regex_replace_all:nnN { \> } { \c{textgreater} } #1
}

\NewDocumentCommand{\change}{m}
{
\tl_set:Nn \l_mynode_text_tl { #1 }
\mynode_replace:N \l_mynode_text_tl
\tl_use:N \l_mynode_text_tl
%\tl_analysis_show:N #1 %%% for debugging
}
\ExplSyntaxOff

\begin{document}

\mynode{root}{Koreň}{root}{gui}

\mynode{gui}{Grafické rozhranie}{root}{nice_gui;fancy_gui}

\mynode{nice_gui}{Pekné grafické rozhranie}{gui}{}

\mynode{fancy_gui}{This text contains less and greater characters:<>}{gui}{}

\mynode{<test>}{Some text with \texttt{under_score}}{<test>}{}
\end{document}


• Thank you. I'm new to LaTeX3 syntax, but it seems to work. One problem is, that this syntax uses ~ and spaces differently, so it breaks preprocessor called Vlna. (It inserts non-breakable space (~) after prepositions. In Slovak and Czech languages, you can't put preposition at the end of line, so I have to use it). I set my compile sequence in a way so that main.tex is not processed by Vlna and now everything works. – user224348 Apr 21 '19 at 16:00
• I have tested your provided example, and it breaks, when I put _ into second argument of \mynode. It throws error, missing $inserted and considers rest of line as math. – user224348 Apr 21 '19 at 16:40 • @user224348 I assumed the text in the second argument was free of underscores. Fixed. – egreg Apr 21 '19 at 16:44 Actually, you have two separate problems: one with font encoding and one with text encoding. Unfortunately, I can help you only with the second one. \detokenize is not UTF8-friendly, but sometimes you can replace it with the following command: \newcommand{\utffriendlydetokenize}[1]{% \scantokens{% \catcode\_=12% \catcode\^=12% \catcode\{=12% \catcode\}=12% \catcode\&=12% \catcode\$=12%
\catcode\#=12%
\catcode\~=12%
\catcode\\=12%
#1%
}%
}


In your case the only row that should be changed is:

\paragraph[\detokenize{#1}]{\utffriendlydetokenize{#2} (\texttt{\detokenize{#1}}):}


And you get your ň back:

However, it does not fix the problem with <>_, because the problem is caused by the codes of the symbols in specified fontenc. (See also Strange behaviour with \detokenize and underscore) The only additional solution that can guess about is to redefine _<> to be active characters by playing with their catcodes, to map them to the corresponding commands and to correct the definition \utffriendlydetokenize for the chosen catcode`s.