Accepting special characters in arguments passing them further

Question

I would like to define a macro very similar to \url, that accepts more or less arbitrary strings (including hashes/#) and passes them to other commands (i.e., \href, \url, \nolinkurl). I have tried to apply some other solutions (e.g., changing catcodes locally etc.), but failed. I would also like my new macro to be passable to other macros like \footnote.

The following non-working MWE shows an implementation, taking arbitrary strings as inputs, converts them to escaped tex-compatible strings (by use of \tl_to_str:n as an xparse command processor), and passes them along without errors. However, this completely breaks the meaning of the macros, which just output the escaped text (I don't really understand why).

Ideally, the macro would even detect hashes and escape them when passing the respective argument to \href, because that's required (but not for the original \url).

\documentclass{article}
\PassOptionsToPackage{hyphens}{url}
\usepackage[hidelinks]{hyperref}

\usepackage{xparse}
\usepackage{letltxmacro}

\ExplSyntaxOn
\LetLtxMacro\oldurl\url
\RenewDocumentCommand\url{>{\tl_to_str:n}m>{\tl_to_str:n}g}{%
    \IfNoValueTF{#2}{%
      \oldurl{#1}%
    }{%
      \href{#1}{#2}%
    }%
}
\ExplSyntaxOff

\setlength{\parindent}{0pt}
\begin{document}

\smallskip
\noindent\rule{\linewidth}{1ex}

\smallskip
url\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break/#me}

\smallskip
2url\#1\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break-me/#please}{www.yahoo.com/break/me/please}

\smallskip
2url\#2\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break-me/please}{www.yahoo.com/break/me/#please}

\smallskip
2url\#3\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break-me/#please}{www.yahoo.com/break-me/#please}

\noindent\rule{\linewidth}{1ex}

footnote: \footnote{\url{https://www.yahoo.com/break-me/#please}{www.yahoo.com/break-me/#please}}
\end{document}

Answer 1

You can't place arbitrary commands in argument processors. An argument processor must leave its results in \ProcessedArgument, so instead of:

\RenewDocumentCommand\url{>{\tl_to_str:n}m}{...}

you'd need:

\cs_new_protected:Npn \Detokenize #1
  { \tl_set:Nx \ProcessedArgument { \tl_to_str:n {#1} } }
\RenewDocumentCommand\url{>{\Detokenize}m}{...}

That said, \detokenize doubles the hashes, so you need to un-double them to get the right URL. I defined an \UndoubleHashes argument processor which turns the argument into a string with \tl_to_str:n, and then loops through it looking for double hashes and replacing them by single ones.

Also, you have to make sure that special characters have the catcode that \url expects, so you need to make \^^M, \%, \#, \&, \_, \~, \$ active (but “other” would also work).

With this change, the tooltip of the PDF viewer shows the correct URL:

\documentclass{article}
\PassOptionsToPackage{hyphens}{url}
\usepackage[hidelinks]{hyperref}

\usepackage{xparse}
\usepackage{letltxmacro}

\makeatletter
\DeclareRobustCommand*{\breakhref}[3][]{%
  \begingroup
    \setkeys{href}{#1}%
    \hyper@normalise\breakhref@{#2}{#3}}
\begingroup
  \lccode`\~=`\#
  \lowercase{\endgroup
  \def\breakhref@#1{\expandafter\breakhref@split#1~~\\}%
  \def\breakhref@split#1~#2~#3\\#4{%
    \hyper@@link{#1}{#2}{#4}%
    \endgroup}}%
\makeatother
\ExplSyntaxOn
\cs_new_protected:Npn \UndoubleHashes #1
  {
    \tl_set:Nx \ProcessedArgument
      { \stefanct_undouble_hashes:n {#1} }
  }
\cs_new:Npx \stefanct_undouble_hashes:n #1
  {
    \exp_not:N \exp_last_unbraced:No
      \exp_not:N \__stefanct_undouble_hashes:w { \exp_not:N \tl_to_str:n {#1} }
        \c_hash_str \c_hash_str \exp_not:N \q_nil \s_stop
  }
\use:x
  {
    \cs_new:Npn \exp_not:N \__stefanct_undouble_hashes:w
        ##1 \c_hash_str \c_hash_str ##2
      {
        ##1
        \exp_not:N \quark_if_nil:nTF {##2}
          { \exp_not:N \use_none_delimit_by_s_stop:w }
          { \c_hash_str ##2 }
        \exp_not:N \__stefanct_undouble_hashes:w
      }
  }

\tl_new:N \l__stefanct_tmpa_tl
\cs_generate_variant:Nn \tl_replace_all:Nnn { Nx }
\LetLtxMacro\oldurl\url
\RenewDocumentCommand\url { }
  {
    \group_begin:
      \tl_map_inline:nn { \^^M \% \# \& \_ \~ \$ }
        { \char_set_catcode_active:n { `##1 } }
      \char_set_catcode_other:n { `\# }
      \urlaux
  }
\group_begin:
\char_set_catcode_active:n { `\# }
\char_set_catcode_parameter:n { `\$ }
\cs_new_protected:Npn \stefanct_url:nn $1 $2
  {
    \tl_set:Nn \l__stefanct_tmpa_tl {$2}
    \tl_replace_all:Nxn \l__stefanct_tmpa_tl { \c_hash_str } { # }
    \char_set_active_eq:NN # \__stefanct_breakable_hash:
    \IfNoValueTF {$2}
      { \oldurl {$1} }
      { \exp_args:NnV \breakhref {$1} \l__stefanct_tmpa_tl }
    \group_end:
  }
\group_end:
\cs_new_protected:Npn \__stefanct_breakable_hash:
  {
    \penalty \UrlBreakPenalty
    \c_hash_str
  }
\NewDocumentCommand\urlaux{>{\UndoubleHashes}m>{\UndoubleHashes}g}
  { \stefanct_url:nn {#1} {#2} }
\ExplSyntaxOff

\setlength{\parindent}{0pt}
\begin{document}

\smallskip
\noindent\rule{\linewidth}{1ex}

\smallskip
url\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break/#me%too}

\smallskip
url\hfill\hspace{0.6\linewidth}\url{httpswwwyahoocombreak#me%too}

\smallskip
2url\#1\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break-me/#please}{www.yahoo.com/break/me/please}

\smallskip
2url\#2\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break-me/please}{www.yahoo.com/break/me/#please}

\smallskip
2url\#3\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break-me/#please}{www.yahoo.com/break-me/#please}

\noindent\rule{\linewidth}{1ex}

footnote: \footnote{\url{https://www.yahoo.com/break-me/#please}{www.yahoo.com/break-me/#please}}
\end{document}

The code allows breaking the URL before # signs. To have it break after a #, change the order of \penalty \UrlBreakPenalty and \c_hash_str in the definition of \__stefanct_breakable_hash:.