# Make ascii string from utf8 input

I want to define a command that will take a name and create a readable string from it, that can be used as a filename. Something like

John Doe -> doe-john.

However, I want to avoid special characters, i.e. replace non-ascii characters by their ascii equivalent. That is, I need a macro that will remove accents and replace some special letters, e.g.

æüßéñ -> aeuessen.

Is there a way to do this in LaTeX with reasonable effort?

• the amount of effort really is proportional to how much input you want to support. it's easy enough to define a function that allows you to specify what to do with each character, but then do you want to map a few dozen european accented characters, or also transliterate cyrillic and greek and a few thousand asian characters and ... Sep 25, 2017 at 21:58
• @DavidCarlisle, I don't need to go beyond european alphabets, but I feel like that list is quite long as it is. I was hoping for some way to just strip any accents off a given character (without needing to resort to a list of every possibility). If I need to consider replacements like umlauts separately (with a list), that's fine, i guess. Sep 25, 2017 at 22:03
• unicode isn't organised in a way that makes stripping accents possible other than a lookup table (although of course in some languages that table may be in some standard string library, but tex you need to do it by hand) I see egreg's posted something (not how I'd have done it but either way you need to make a long list of replacements) Sep 25, 2017 at 22:05
• If you're using inputenc (rather than XeLaTeX/LuaLaTeX) then you could let \IeC and all the accent commands to \@firstofone. (The glossaries package's \glsnoidxstripaccents does this to strip accents from the sort values when using TeX to sort.) Sep 25, 2017 at 22:53
• @NicolaTalbot that's not a bad idea, you should post an answer, it may prevent egreg getting a tick, which is always a good thing. Sep 26, 2017 at 7:56

If you are using inputenc (rather than XeLaTeX/LuaLaTeX) you can take advantage of the fact that inputenc turns the extended characters into accent commands. For example, ü expands to \IeC{\"u}. So you can temporarily redefine the accent commands to strip them out.

Example:

\documentclass{article}
\usepackage[utf8]{inputenc}

\makeatletter
\newcommand{\stripaccents}[2]{%
\begingroup
% strip accents:
% provide replacement strings:
\def\AE{AE}%
\def\ae{ae}%
\def\OE{OE}%
\def\oe{oe}%
\def\AA{AA}%
\def\aa{aa}%
\def\L{L}%
\def\l{l}%
\def\O{O}%
\def\o{o}%
\def\SS{SS}%
\def\ss{ss}%
\def\th{th}%
\def\TH{TH}%
\def\dh{dh}%
\def\DH{DH}%
\xdef#1{#2}%
\endgroup
}
\makeatother

\begin{document}
\stripaccents\tmp{æüßéñ}
\show\tmp
\end{document}


This shows:

> \tmp=macro:
->aeussen.


If you have any other commands that are likely to occur in your input, you'll need to add them to \stripaccents so that they expand into something sensible.

For the umlauts, you could temporarily redefine \" so that it appends e to its argument:

\newcommand{\stripaccents}[2]{%
\begingroup
\def\"##1{##1e}% umlaut
\def\AE{AE}%
\def\ae{ae}%
\def\OE{OE}%
\def\oe{oe}%
\def\AA{AA}%
\def\aa{aa}%
\def\L{L}%
\def\l{l}%
\def\O{O}%
\def\o{o}%
\def\SS{SS}%
\def\ss{ss}%
\def\th{th}%
\def\TH{TH}%
\def\dh{dh}%
\def\DH{DH}%
\xdef#1{#2}%
\endgroup
}


This now shows:

> \tmp=macro:
->aeuessen.


With T1 encoding you also need:

\let\@text@composite@x\@secondoftwo


in the definition of \stripaccents, as mentioned in your comment.

• This is what I was hoping for! I am also using \usepackage[T1]{fontenc} which breaks this solution. However, adding \let\@text@composite@x\@secondoftwo to the definition of \stripaccents rectifies this. May be worth adding to the answer. Sep 26, 2017 at 15:34
• @schtandard Sorry, I should have checked with T1. I'll add it in. Sep 26, 2017 at 15:39
• One more thing I noticed: fontenc expands å to \r a, not \aa. Thus, instead of \AA and \aa we have to redefine \r. Sep 26, 2017 at 15:46
• @schtandard Perhaps the best definition would be \def\r##1{##1##1} assuming that the ring accent indicates a long vowel. (If not, then a more complex definition would be required that expands the argument to something appropriate. For example, if \r a needs to be aa but \r y needs to be uo.) Sep 26, 2017 at 15:56
• agreed. \def\r##1{##1##1} should be sufficient for my needs. Sep 26, 2017 at 17:07

You have to populate the list yourself, according to the given examples.

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn
\seq_new:N \g_schtandard_search_replace_seq

\seq_gput_right:Nn \g_schtandard_search_replace_seq { {æ}{ae} }
\seq_gput_right:Nn \g_schtandard_search_replace_seq { {ä}{ae} }
\seq_gput_right:Nn \g_schtandard_search_replace_seq { {ö}{oe} }
\seq_gput_right:Nn \g_schtandard_search_replace_seq { {ü}{ue} }
\seq_gput_right:Nn \g_schtandard_search_replace_seq { {ß}{ss} }
\seq_gput_right:Nn \g_schtandard_search_replace_seq { {ñ}{n} }
\seq_gput_right:Nn \g_schtandard_search_replace_seq { {é}{e} }

\tl_new:N \l_schtandard_input_tl

\NewDocumentCommand{\makestring}{om}
{
\tl_set:Nn \l_schtandard_input_tl { #2 }
\seq_map_inline:Nn \g_schtandard_search_replace_seq
{
\regex_replace_all:nnN ##1 \l_schtandard_input_tl
}
\IfNoValueTF{#1}
{
\tl_use:N \l_schtandard_input_tl
}
{
\tl_set_eq:NN #1 \l_schtandard_input_tl
}
}
\ExplSyntaxOff

\begin{document}

\makestring{æüßéñ}

\makestring[\foo]{æüßéñ}\texttt{\meaning\foo}

\end{document}


• And there I thought the days of me not knowing most of the commands in an answer where over... Guess it's time to learn expl3. Sep 25, 2017 at 22:15

If you only need filenames, but do not need them to be "human readable", then you could take advantage of \pdfstringdef

\documentclass{article}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[unicode]{hyperref}

\makeatletter
\begingroup
\catcode| 0 \catcode\\ 12
|gdef|makestring@i\#1#2#3#4%
{#1#2#3|if|relax#4|expandafter|@gobbletwo|fi|makestring@i#4}
|endgroup
\newcommand*{\makestring}[2]{%
\pdfstringdef\makestring@{#2}%
\edef#1{\expandafter\makestring@i\makestring@\relax}%
}
\makeatother

\begin{document}

\makestring{\foo}{æüßéñ}

\texttt{\meaning\foo}

\end{document}


A variation on this theme which is much more efficient, it show the utf8 bytes. One could produce in hexadecimal if desired. (in fact there are possibly macros in utf8.def which could be used here)

\documentclass{article}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}

\makeatletter
\newcommand*\MakeString[2]{%
\begingroup
\def\UTFviii@two@octets##1##2{\the\numexpr##1\relax\the\numexpr##2}%
\def\UTFviii@three@octets##1##2##3{\the\numexpr##1\relax\the\numexpr##2\relax\the\numexpr##3\relax}%
\def\UTFviii@four@octets##1##2##3##4{\the\numexpr##1\relax\the\numexpr##2\relax\the\numexpr##3\relax\the\numexpr##4\relax}%
\xdef#1{#2}%
\endgroup
}
\makeatother

\begin{document}

\MakeString{\foo}{æüßéñ}

\texttt{\meaning\foo}

\show\foo
\end{document}


Produces:

> \foo=macro:
->195166195188195159195169195177.
l.23 \show\foo


I should improve so that each byte produce a three-digits decimal, here leading zeros are stripped!

Ok here it is with no stripping and 2-hex digits per byte.

edit removed usage of extra package. Defined \Byte@tohex macro possibly already provided by utf8-inputenc internally, not checked.

\documentclass{article}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}

\makeatletter
% I have not checked but maybe utf8-inputenc provides already
% similar macro (not even using e-TeX)
\def\Byte@tohex #1%
{\expandafter
\Byte@tohex@\the\numexpr(#1+8)/16-1\expandafter
.\the\numexpr#1.}%
\def\Byte@tohex@ #1.#2.%
{\Byte@onehex #1.%
\expandafter\Byte@onehex\the\numexpr #2-16*#1.%
}
\def\Byte@onehex #1.%
{\ifcase #1
0\or1\or2\or3\or4\or5\or6\or7\or8\or9%
\or A\or B\or C\or D\or E\or F%
\fi
}%
\newcommand*\MakeString[2]{%
\begingroup
\def\UTFviii@two@octets##1##2{\Byte@tohex{##1}\Byte@tohex{##2}}%
\def\UTFviii@three@octets##1##2##3{\Byte@tohex{##1}\Byte@tohex{##2}\Byte@tohex{##3}}%
\def\UTFviii@four@octets##1##2##3##4{\Byte@tohex{##1}\Byte@tohex{##2}\Byte@tohex{##3}\Byte@tohex{##4}}%
\xdef#1{#2}%
\endgroup
}
\makeatother
\begin{document}

\MakeString{\foo}{æüßéñ}

\texttt{\meaning\foo}

\show\foo
\end{document}


produces in log

> \foo=macro:
->C3A6C3BCC39FC3A9C3B1.
l.27 \show\foo

• I see now OP said create a readable string but I am not sure if that meant create a recognizable string this is why I am posting this nevertheless.
– user4686
Sep 26, 2017 at 8:53
• of course this is also a recognizable string for the happy ones versed in octal sequences and UTF-16BE.
– user4686
Sep 26, 2017 at 8:58
• @StevenB.Segletes I am only the lazy guy... I should rather imitate utf8-inputenc but use it to produce an octal or hex sequence straight from the utf8-encoded source. The \pdfstringdef is rather big macro and makes many things like so needed here. But it is time for Lunch!
– user4686
Sep 26, 2017 at 10:13
• "Choose a Lazy Person To Do a Hard Job Because That Person Will Find an Easy Way To Do It" (quoteinvestigator.com/2014/02/26/lazy-job) Sep 26, 2017 at 11:44
• Why not just \pdfescapehex the detokenized version of the string? \message{\pdfescapehex\expandafter{\detokenize{éçß☺}}}` Sep 26, 2017 at 16:46