3

A way to get the LICR representation of a string (in pdflatex) is with \protected@edef. For example:

\documentclass{article}

\usepackage[cp1252]{inputenc}

\begin{document}

\makeatletter
\protected@edef\result{^^f6}
\show\result

\end{document}

prints

> \result=macro:
->\"o.

So far so good.

Formerly, this used to work if the encoding was utf8. With TL2018:

\documentclass{article}

\usepackage[utf8]{inputenc}

\begin{document}

\makeatletter
\protected@edef\result{^^c3^^b6} % ie, ^^f6 in UTF-8 encoding
\show\result

\end{document}

the following was shown:

> \result=macro:
->\IeC {\"o}.

But I’ve just realized it no longer works, because now it returns the actual character:

> \result=macro:
->ö.

This means the resulting string can’t be manipulated in an encoding independent way. The reason for this change in behavior is the characters are now \protected (an eTeX primitive), thus preventing the expansion to their internal definition, aka LICR.

  • Is there a way to make them un-\protected without redefining the characters every time a LICR is to be generated?
  • Or more in general, is there an alternative way to generate the LICR representation with utf8, as with cp1252, applemac and other 8-bit encodings.
6
  • I'm not sure what you mean by ' string can’t be manipulated in an encoding independent way': the idea is that everything converts to UTF-8, and then internally you always use that.
    – Joseph Wright
    Feb 28 at 17:15
  • @JosephWright But this is not what 8-bit encodings are doing. Feb 28 at 17:20
  • Sure, you always cover the UTF-8 and macro version, as even in UTF-8 the user might have \"o or ö, and that difference should be retained as far as possible.
    – Joseph Wright
    Feb 28 at 17:24
  • @JosephWright The macro will print a wrong result after a \inputencoding{cp1252}. Well, admittedly this is a quite infrequent situation, but formerly it worked. Feb 28 at 17:39
  • Something similar was discussed on the team list when we made the change: you can't for example 'know' the right answer for \def\foo{ö}\inputencoding{cp1252}\protected@edef\foo{\foo}. So the conclusion was that it's not feasible to make the change (needed for other reasons) and keep 100% the same behaviour.
    – Joseph Wright
    Feb 28 at 18:19

3 Answers 3

4

Define \getlicr in the following way:

\documentclass{article}

\ExplSyntaxOn
\NewDocumentCommand{\getlicr}{mm}{%
  \cs_set_eq:Nc #1 { u8:\tl_to_str:n {#2} }
}
\ExplSyntaxOff

\getlicr{\result}{^^c3^^b6}
\show\result

\getlicr{\result}{ö}
\show\result

\stop

Console output:

> \result=macro:
->\IeC {\"o}.
l.10 \show\result

?
> \result=macro:
->\IeC {\"o}.
l.13 \show\result

With some stretches…

\documentclass{article}

\ExplSyntaxOn
\NewDocumentCommand{\getlicr}{mm}
 {
  \javier_getlicr:nn { #1 } { #2 }
 }

\tl_new:N \g__javier_licr_item_tl
\seq_new:N \l__javier_licr_in_seq
\seq_new:N \l__javier_licr_out_seq

\cs_new_protected:Nn \javier_getlicr:nn
 {
  \seq_set_split:Nnn \l__javier_licr_in_seq { ~ } { #2 }
  \seq_clear:N \l__javier_licr_out_seq
  \seq_map_function:NN \l__javier_licr_in_seq \__javier_licr:n
  \tl_set:Ne #1 { \seq_use:Nn \l__javier_licr_out_seq { ~ } }
 }

\cs_new_protected:Nn \__javier_licr:n
 {
  \tl_gclear:N \g__javier_licr_item_tl
  \text_map_function:nN { #1 } \__javier_licr_item:n
  \group_begin:
    \cs_set:Npn \IeC ##1 { \exp_not:n { ##1 } }
    \tl_gset:Ne \g__javier_licr_item_tl { \g__javier_licr_item_tl }
  \group_end:
  \seq_put_right:NV \l__javier_licr_out_seq \g__javier_licr_item_tl
}

\cs_new_protected:Nn \__javier_licr_item:n 
 {
  \tl_if_single:nTF { #1 }
   {
    \tl_gput_right:Nn \g__javier_licr_item_tl {#1}
   }
   {
    \tl_gput_right:Nv \g__javier_licr_item_tl { u8:\tl_to_str:n {#1} }
   }
 }
\ExplSyntaxOff

\begin{document}

\getlicr{\result}{^^c3^^b6}
\texttt{\meaning\result} $\to$ \result

\getlicr{\result}{eäñçîw xyž}
\texttt{\meaning\result} $\to$ \result

\end{document}

enter image description here

4
  • Nice, and upvoted, but it converts just a character, not a string. Feb 28 at 17:29
  • @JavierBezos I was adding it…
    – egreg
    Feb 28 at 17:37
  • Fails if there is space, but anyway you have put me on the right track, so I’ll accept it. Feb 28 at 17:47
  • @JavierBezos Reworked for spaces
    – egreg
    Feb 28 at 18:06
4

You aren't supposed to need this:-) but...

\documentclass{article}


\usepackage[utf8]{inputenc}

\begin{document}


\makeatletter
{
  \def\IeC#1{\unexpanded{\IeC{#1}}}
\expandafter
\edef
\expandafter
\result
\expandafter
{^^c3^^b6} % ie, ^^f6 in UTF-8 encoding
\show\result
}

\stop

produces

> \result=macro:
->\IeC {\"o}.
l.18 \show\result
2
  • Yes, I knew the \expandafter trick (when I discovered I was perplexed), but sadly it only works for the first character. Feb 28 at 17:18
  • 1
    @JavierBezos well yes etex \protected has many benefits over latex \protect but the one disadvantage is that it's a lot harder to locally undo. The l3 text module will probably do what you want (it knows how to decompose these definitions) but it depends what you really want Feb 28 at 17:28
3

I suspect you are after \text_purify:n:

\documentclass{article}

\usepackage[cp1252]{inputenc}
\ExplSyntaxOn
\let\expander\text_purify:n
\ExplSyntaxOff

\begin{document}

\makeatletter
\edef\result{\expander{^^f6}}
\show\result

\end{document}

This should give a UTF-8 representation as far as possible for most input encodings (at some stage it likely needs extending to cover all of at least PU).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .