2

I want to store text that contains non-ascii characters in latex3 string variables. With xelatex and lualatex it works well but With pdflatex next mwe produces incorrect output. For legacy reasons switching to xelatex or lualatex isn't that easy on the project that I'm working with, so would it be possible to make next mwe work correctly?

MWE:

\documentclass{article}

\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{xparse}

\ExplSyntaxOn

\NewDocumentCommand{\savestring}{mm}{%
    \str_clear_new:c{g_mwe_#1_str}
    \str_gset:cn{g_mwe_#1_str}{#2}
}

\NewDocumentCommand{\printstring}{m}{%
    \str_use:c{g_mwe_#1_str}
}

\ExplSyntaxOff


\begin{document}
\savestring{umlaut}{hääyöaie}

Hääyöaie

\printstring{umlaut}
\end{document}

Result is:

Result from mwe

  • 1
    You can't store UTF-8 characters as strings, as you discovered. Why not token lists? – egreg Jul 8 '18 at 18:50
  • 1
    latex3 strings are catcode12 characters (like punctuation) and spaces. inputenc needs active (catcode13) characters to decode UTF-8. – David Carlisle Jul 8 '18 at 19:06
  • There's no problem if you use token lists - I do this all the time. (And I only use pdfTeX.) – cfr Jul 8 '18 at 21:18
3

Strings are “detokenized”: every character in them has category code 12, except for spaces that have catcode 10.

You can rescan the string:

\documentclass{article}

\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{xparse}

\ExplSyntaxOn

\NewDocumentCommand{\savestring}{mm}
 {
    \str_clear_new:c{g_mwe_#1_str}
    \str_gset:cn{g_mwe_#1_str}{#2}
 }

\NewDocumentCommand{\printstring}{m}
 {
    \tl_rescan:nv {} {g_mwe_#1_str}
 }
\cs_generate_variant:Nn \tl_rescan:nn { nv }

\ExplSyntaxOff


\begin{document}
\savestring{umlaut}{hääyöaie}

Hääyöaie

\printstring{umlaut}
\end{document}

but what you really want is saving the tokens in token list variables.

\documentclass{article}

\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{xparse}

\ExplSyntaxOn

\NewDocumentCommand{\savestring}{mm}
 {
    \tl_clear_new:c {g_mwe_#1_tl}
    \tl_gset:cn {g_mwe_#1_tl}{#2}
 }

\NewDocumentCommand{\printstring}{m}
 {
    \tl_use:c {g_mwe_#1_tl}
 }

\ExplSyntaxOff


\begin{document}
\savestring{umlaut}{hääyöaie}

Hääyöaie

\printstring{umlaut}
\end{document}

enter image description here

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.