Why does this protection not work?

Question

The answers to this question explain in detail what protection is so I'm not wondering about that. My question is why a certain command definition with expl3 does not give a protected command.

Consider this LaTeX2e example:

\documentclass{article}

\DeclareRobustCommand\foo[1]{#1}
\DeclareRobustCommand\?{??}
\expandafter\DeclareRobustCommand\expandafter\-\expandafter{\-}

\begin{document}

\tableofcontents
% this is <jobname>.toc:
% \contentsline {section}{\numberline {1}\foo {bar}}{1}
% \contentsline {section}{\numberline {2}foo\?}{1}
% \contentsline {section}{\numberline {3}foo\-bar}{1}

\section{\foo{bar}}

\section{foo\?}

\section{foo\-bar}

\end{document}

Everything is working as expected.

However, when I try to do the same with expl3 the \- is not protected:

\documentclass{article}

\usepackage{expl3}
\ExplSyntaxOn
\cs_new_protected:Npn \foo #1 { #1 }          % works
\cs_new_protected:Npn \? { ?? }               % works
\exp_args:NNo \cs_set_protected:Npn \- { \- } % does not work
\ExplSyntaxOff

\begin{document}

\tableofcontents
% this is <jobname>.toc:
% \contentsline {section}{\numberline {1}\foo {bar}}{1}
% \contentsline {section}{\numberline {2}foo\?}{1}
% \contentsline {section}{\numberline {3}foo\discretionary {-}{}{}bar}{1}
% \contentsline {section}{\numberline {4}foo\-bar}{1}

\section{\foo{bar}}

\section{foo\?}

\section{foo\-bar}

\section{foo\protect\-bar}

\end{document}

Am I doing something wrong or why does \exp_args:NNo \cs_set_protected:Npn \- { \- } not leave the desired result? The only command where I could find this behaviour is \-. Did I not look hard enough or is there something special to \-?

Answer 1

You are being caught out not by expl3 but by LaTeX2e! The redefinition you use is correct, which you will see if you \show\-:

> \-=\protected\long macro:
->\discretionary {-}{}{}.

However, LaTeX2e redefines \- during the output routine, which is where the .toc entries are created:

\@arrayparboxrestore ->\let \if@nobreak \iffalse \let \if@noskipsec \iffalse \l
et \par \@@par \let \-\@dischyph \let \'\@acci \let \`\@accii \let \=\@acciii \
parindent \z@ \parskip \z@skip \everypar {}\linewidth \hsize \@totalleftmargin 
\z@ \leftskip \z@skip \rightskip \z@skip \@rightskip \z@skip \parfillskip \@flu
shglue \lineskip \normallineskip \baselineskip \normalbaselineskip \sloppy 
{\let}
{reassigning \if@nobreak=\iffalse}
{\let}
{reassigning \if@noskipsec=\iffalse}
{\let}
{reassigning \par=\par}
{\let}
{changing \-=\protected\long macro:->\discretionary {-}{}{}}
{into \-=macro:->\discretionary {-}{}{}}
...

So you may wonder why we see a difference between the behaviour with a protected command and a LaTeX2e 'robust' one: after all, both have been ignored by the output routine! It comes down to how the different protection works. With the LaTeX3 method, it's down to the engine, so each time an \edef or similar is applied to \- it stays totally unchanged. However, the protection is removed by LaTeX2e.

On the other hand, the LaTeX2e robust mechanism results in \- turning into \protect\- inside an \edef. That means that even if we change the definition of \- later, the protection 'hangs about'.

You see this where TeX writes to the .toc when \tracingall. With LaTeX2e's robust mechanism:

\write->\@writefile{toc}{\protect \contentsline {section}{\protect \numberline 
{1}foo\protect \-bar}{\thepage }}

but with LaTeX3's native engine system

\write->\@writefile{toc}{\protect \contentsline {section}{\protect \numberline 
{1}foo\-bar}{\thepage }}

As egreg has already commented, the best solution at the moment is to make \@dischyph engine-protected by setting it equal to \-. (Of course, longer-term a LaTeX3 format will have everything protected or expandable, and this sort of issue should go away.)

---

By the way, I'd use

\cs_set_protected_nopar:Npx \- { \exp_not:o { \- } }

Answer 2

Among what LaTeX does during the output routine there is \@arrayparboxrestore which emits, among other things,

\-\@dischyph

where \@dischyph is defined by the kernel with

560 \def\-{\discretionary{-}{}{}}
561 \let\@dischyph=\-

(I've left the line numbers). This means that when the .aux file is written out, \- has again the value it had before your redefinition (and is not protected, of course). With \DeclareRobustCommand what's passed to the .aux file is not \-, but something different (it's really \protect\- that, since \protect is \noexpand, becomes the unexpanded \-). Thus the problem does not show.

Solution: add

\makeatletter
\let\@dischyph\-
\makeatother

after having redefined \-.