# How to make \tl_upper_case:n applied to diacritics work in section argument?

The LaTeX3 Interfaces' documentation claims, page 224:

In package mode, the case change system will also convert text stored using the LaTeX 2ε “LICR” approach. This will upper/lower case tokens as implemented for the font encodings T1, T2, T5 and LGR (see the behaviour of the LaTeX 2ε command \MakeUppercase). Note that these commands will automatically be protected from expansion.

I don't pretend I fully understand all of this (what's exactly the "package mode", encodings, etc.) but I notice significant different behavior between LaTeX3 case change system and the LaTeX 2ε command \MakeUppercase, as shown by the following:

\documentclass{article}
\usepackage{ifxetex}
\usepackage{ifluatex}
\ifluatex
\usepackage{fontspec}
\else
\ifxetex
\usepackage{fontspec}
\else
\usepackage[utf8]{inputenc}\usepackage[T1]{fontenc}
\fi
\fi
\usepackage{lmodern}
\usepackage{xparse}
\begin{document}
\ExplSyntaxOn
\begin{enumerate}
\item{\ttfamily\tl_to_str:n {\MakeUppercase{aäa}}}~$\to$~\MakeUppercase{aäa}:
\begin{description}
\item[pdflatex:] OK.
\item[xelatex/lualatex:] OK.
\end{description}
\item{\ttfamily\tl_to_str:n {\MakeUppercase{a\"aa}}}~$\to$~\MakeUppercase{a\"aa}:
\begin{description}
\item[pdflatex:] OK.
\item[xelatex/lualatex:] OK.
\end{description}
\item{\ttfamily\tl_to_str:n
{\tl_upper_case:n{aäa}}}~$\to$~%\tl_upper_case:n{aäa}:
\begin{description}
\item[pdflatex:] Fails~("AäA")~with~a~not~critical~error:
\ExplSyntaxOff
\begin{verbatim}
! Missing \endcsname inserted.
\protect
l.29 ...ase:n{aäa}}}~$\to$~\tl_upper_case:n{aäa}
:
?
\end{verbatim}
\ExplSyntaxOn
\item[xelatex/lualatex:] OK.
\end{description}
\item{\ttfamily\tl_to_str:n {\tl_upper_case:n {a\"aa}}}~$\to$~\tl_upper_case:n{a\"aa}:
\begin{description}
\item[pdflatex:] Fails~("AäA")~without~any~error.
\item[xelatex/lualatex:] Fails~("AäA")~without~any~error.
\end{description}
\end{enumerate}

\section{
{\ttfamily\tl_to_str:n {\MakeUppercase{aäa}}}~$\to$~\MakeUppercase{aäa}%
}
\begin{description}
\item[pdflatex:] OK.
\item[xelatex/lualatex:] OK.
\end{description}
\section{
{\ttfamily\tl_to_str:n {\MakeUppercase{a\"aa}}}~$\to$~\MakeUppercase{a\"aa}
}
\begin{description}
\item[pdflatex:] OK.
\item[xelatex/lualatex:] OK.
\end{description}
\section{
{\ttfamily\tl_to_str:n {\tl_upper_case:n {aäa}}}~$\to$~
% \tl_upper_case:n{aäa}
}
\begin{description}
\item[pdflatex:] Fails~("AäA")~with~a~not~critical~error:
\ExplSyntaxOff
\begin{verbatim}
! Missing \endcsname inserted.
\protect
l.71 }

?
\end{verbatim}
\ExplSyntaxOn
\item[xelatex/lualatex:] OK.
\end{description}
\section{
{\ttfamily\tl_to_str:n {\tl_upper_case:n {a\"aa}}}~$\to$~
% \tl_upper_case:n{a\"aa}
}
\begin{description}
\item[pdflatex:] Fails~with~lots~of~critical~errors:
\ExplSyntaxOff
\begin{verbatim}
! Interruption.
\__tl_change_case_N_type:Nnnn ...q_recursion_stop
{#3}{#4}
l.89 }

?
[...]
\end{verbatim}
\ExplSyntaxOn
\item[xelatex:] Fails~with~lots~of~critical~errors:
\ExplSyntaxOff
\begin{verbatim}
! Interruption.
\q_nil
l.87 }

?
^C! Interruption.
**\prg_return_true: \else:...
l.87 }

?
[...]
\end{verbatim}
\ExplSyntaxOn
\item[lualatex:] Fails~with~lots~of~critical~errors:
\ExplSyntaxOff
\begin{verbatim}
! Interruption.
\q_nil
l.87 }

?
^C! Interruption.
\__str_escape_x:n ...er:wN {\luatex_expanded:D {#1
}}}
l.87 }

?
[...]
\end{verbatim}
\ExplSyntaxOn
\end{description}
\ExplSyntaxOff
\end{document}


Could you explain these different behaviors?

In particular, I'd like to get \tl_upper_case:n{aäa} or, better, \tl_upper_case:n{a\"aa} (because will be in a class file) to work in \section argument with pdflatex: is it possible?

• I can explain (after work: going to be long) but this might be one for LaTeX-L as it partly is about what is 'realistic'/'useful'/'needed'/'maintainable'. Jan 8 '16 at 14:00

The expandable case changing code now has a final 'home' in expl3 as \text_uppercase:n:

\documentclass{article}
\usepackage{lmodern}        % Over-ridden by fontspec for XeTeX/LuaTeX
\usepackage[T1]{fontenc}    % Over-ridden by fontspec for XeTeX/LuaTeX
\ifnum 0%
\ifdefined\XeTeXversion 1\fi
\ifdefined\directlua    1\fi
>0 %
\usepackage{fontspec}
\fi

\usepackage{xparse}
\ExplSyntaxOn
\DeclareExpandableDocumentCommand \ExplUpperCase { m } { \text_uppercase:n {#1} }
\ExplSyntaxOff

\NewDocumentCommand\demo{m}{\texttt{\detokenize{#1}} $\to$ #1}
\begin{document}

\section{\demo{\MakeUppercase{aäa}}}

\section{\demo{\MakeUppercase{a\"aa}}}

\section{\demo{\ExplUpperCase{aäa}}}

\section{\demo{\ExplUpperCase{a\"aa}}}

\end{document}


The code here is very much experimental and subject to change/improvement. In particular, I've now revised the coverage of UTF-8 input for pdfTeX and accents more generally. With the latest expl3 the following will work without issue.

\documentclass{article}
\usepackage{lmodern}        % Over-ridden by fontspec for XeTeX/LuaTeX
\usepackage[T1]{fontenc}    % Over-ridden by fontspec for XeTeX/LuaTeX
\usepackage[utf8]{inputenc} % This bails out for XeTeX/LuaTeX
\ifnum 0%
\ifdefined\XeTeXversion 1\fi
\ifdefined\directlua    1\fi
>0 %
\usepackage{fontspec}
\fi

\usepackage{xparse}
\ExplSyntaxOn
\DeclareExpandableDocumentCommand \ExplUpperCase { m } { \tl_upper_case:n {#1} }
\ExplSyntaxOff

\NewDocumentCommand\demo{m}{\texttt{\detokenize{#1}} $\to$ #1}
\begin{document}

\section{\demo{\MakeUppercase{aäa}}}

\section{\demo{\MakeUppercase{a\"aa}}}

\section{\demo{\ExplUpperCase{aäa}}}

\section{\demo{\ExplUpperCase{a\"aa}}}

\end{document}


This code is likely to go to CTAN this week (2016-01-14 or thereabout).

It's important to note that the code here is experimental, and there are still ideas to be tested, developed and discussed. It's also important to note that expl3 is intended as the programming layer for a format to succeed LaTeX2e: as such, there are functions that may work for 'some future format' but which more restricted in the LaTeX2e environment. This leads to the split between 'format mode' (looking to the future) and 'package mode' (when used with LaTeX2e).

The 'design brief' for \tl_upper_case:n is broadly:

• To allow case changing across the Unicode range including handling mappings which are not one-to-one
• Seamlessly deal with context-sensitive conversions where appropriate
• Mechanisms to handle language-dependent conversions
• Allow an escape mechanism including skipping math mode material
• Ideally to work expandably

This is all doable but there are additional questions when dealing with macro-based input and in particular when looking at pdfTeX support. For a Unicode engine with Unicode letter input, the set up is relatively easy but becomes more complex when covering the other cases. At present, the approach taken is to special case a range of LICR inputs (things like \aa) but not accent commands. It also makes no attempt to cover UTF-8 input outside of the 7-bit range with pdfTeX. As such, it's unsurprising (to me) that the above fails.

Note that \MakeUppercase is not expandable, and something like \protected@edef\temp{<input>}\tl_lower_case:V \temp will work with input such as ä with pdfTeX as it gets converted by the LaTeX2e mechanisms input (essentially) \"a.

Upcoming changes in how LaTeX2e approach UTF-8 input coverage with pdfTeX suggest to me that a slightly different formulation is probably better than the current one. I plan to revise the code such that input covered by the T1 font encoding will case-change correctly if the input file is in UTF-8. This may take a few days.

• I guess you meant \tl_upper_case instead of \tl_lower_case and both of them require \cs_generate_variant:Nn \tl_(upp|low)er_case:n { V }. Anyway, both \protected@edef\temp{aäa}\tl_upper_case:V \temp and \protected@edef\temp{AÄA}\tl_lower_case:V \temp don't give the expected results (respectively "AäA" and "aÄa"). Jan 9 '16 at 7:26
• About your updated answer: I was waiting impatiently for the new code each daily TL update and, finally, it came this morning :) Works very nicely! Jan 28 '16 at 7:56