Avoid @ hackery: How can I preserve spaces in a format specification when processing the specification with expl3 and PGF/TikZ?

Question

This is a simplified version of part of the code I've been using in answers based on chronos, an attempt to create something somewhat like a substitute for the problematic chronosys.

This code works as it should but it relies on a weird hack. How should this be done without the need for any weird hackery?

The relevant weird hack (i.e. the one which I know is weird) involves the lines

  \tl_replace_all:Nnn \l_chronos_dateformat_tl { ~ } { @ }

and

      { @ } { ~ }

Essentially what this does is to substitute all spaces in the user-defined date format with @ and then substitute all @s in that date format with spaces when typesetting the formatted date.

The idea is that a user can say something like

\tikzset{%
   date format={ B d, Y},
}

in order for dates of timeline events to be printed as September 15, 2013, for example, or

\tikzset{%
  date format={b d Y},
}

for Sept 15 2013, say. (The minimal code here won't do this because I've removed the code needed for b, B and d to work, but I wanted to explain why I'm going about regurgitating the year the user inputs in such a roundabout way.)

The idea is that any character which is not recognised by the code as a date-format character will simply be passed through unaltered. However, spaces are a natural character to use in certain date formats and so the problem is to preserve those during the manipulation of the remaining characters.

It is also worth saying that the date is not immediately typeset as here but, is rather, stored and output as part of a TikZ node later. This explains some of the complexity (e.g. the use of #4 to create a tag to retrieve the date such as thing) not explained by the formatting itself. (In fact, most of the apparatus is there for this reason and not because of the format-parsing requirement.)

I'm a bit worried that I may not have explained fully enough why I'm doing it this way, so if it seems there is some obvious, simple solution which involves demolishing most of the complexity I've introduced, then I'm certainly interested, but please ask for clarification before spending your time writing an answer!

\documentclass[border=10pt,multi,tikz]{standalone}
\usepackage{xparse}
\makeatletter
\ExplSyntaxOn
\tl_new:N \l_chronos_dateformat_tl
\tl_set:Nn \l_chronos_dateformat_tl { d/m/Y }
\cs_generate_variant:Nn \int_abs:n { c }
\cs_new_protected_nopar:Npn \chronos_show_date:n #1
{
  \tl_map_inline:Nn \l_chronos_dateformat_tl
  {
    \str_case:nnF { ##1 }
    {
      { Y } { \int_abs:c { chronos@#1year } } % for each Y in the date format, output the year in YYYY, omitting any leading minus sign
      { @ } { ~ } % un-hackery: for each @ s in the date format, output a space when showing a date
    }
    {
      ##1
    }
  }
}
\cs_new_protected_nopar:Npn \chronos_set_dateformat:n #1
{
  \tl_set:Nn \l_chronos_dateformat_tl { #1 }
  \tl_replace_all:Nnn \l_chronos_dateformat_tl { ~ } { @ } % hackery - substitute any spaces in the input with @
}
\NewDocumentCommand \chronos@setdateformat { m }
{
  \chronos_set_dateformat:n { #1 }
}
\NewDocumentCommand \chronos@showdate { o m }
{
  \group_begin:
    \IfValueT { #1 }
    {
      \chronos_set_dateformat:n { #1 }
    }
    \chronos_show_date:n { #2 }
  \group_end:
}
\ExplSyntaxOff
\newcounter{chronos@thingdate}
\tikzset{%
  chronos set date/.code args={#1:#2:#3:#4}{%
    \expandafter\def\csname chronos@#4year\endcsname{#1}%
    \chronos@showdate{#4}% This doesn't usually happen here: instead, the date is saved with a tag such as 'thing' so that the date or parts of the date can be retrieved and formatted later, possibly more than once with different formatting. Multiple tags allow the storing of multiple sets of date/date part information. The information gets used in various ways to calculate, construct and annotate a timeline.
  },
  chronos date/.style args={#1-#2-#3}{%
    /tikz/chronos set date/.expanded={#1:#2:#3:thing}% In the full code 'thing' is one of several tags used to tag dates for the timeline.
  },
  date format/.code={% Let the user change the format of the date without doing anything special to include spaces.
    \chronos@setdateformat{#1}%
  },
}
\makeatother
\begin{document}
  \tikzset{%
    date format={date: Y},
    chronos date=2016-04-25,
  }
\end{document}

EDIT

Using Manuel's suggestion (sort of) from the comments, I can do away with the @-ery at the price of somewhat greater complexity. (Note, however, that I'm not at all sure this is what Manuel intended, so the fault is certainly mine!)

\documentclass[border=10pt,multi,tikz]{standalone}
\usepackage{xparse}
\makeatletter
\ExplSyntaxOn
\seq_new:N \l_chronos_date_seq
\seq_new:N \l_chronos_dateformat_seq
\seq_set_split:Nnn \l_chronos_dateformat_seq { ~ } { d/m/Y }
\cs_generate_variant:Nn \int_abs:n { c }
\cs_new_protected_nopar:Npn \chronos_show_date:n #1
{
  \seq_clear:N \l_chronos_date_seq
  \seq_map_inline:Nn \l_chronos_dateformat_seq
  {
    \seq_put_right:Nn \l_chronos_date_seq
    {
      \tl_map_inline:nn { ##1 }
      {
        \str_case:nnF { ####1 }
        {
          { Y } { \int_abs:c { chronos@#1year } }
        }
        {
          ####1
        }
      }
    }
  }
  \seq_use:Nn \l_chronos_date_seq { ~ }
}
\cs_new_protected_nopar:Npn \chronos_set_dateformat:n #1
{
  \seq_set_split:Nnn \l_chronos_dateformat_seq { ~ } { #1 }
}
\NewDocumentCommand \chronos@setdateformat { m }
{
  \chronos_set_dateformat:n { #1 }
}
\NewDocumentCommand \chronos@showdate { o m }
{
  \group_begin:
    \IfValueT { #1 }
    {
      \chronos_set_dateformat:n { #1 }
    }
    \chronos_show_date:n { #2 }
  \group_end:
}
\ExplSyntaxOff
\newcounter{chronos@thingdate}
\tikzset{%
  chronos set date/.code args={#1:#2:#3:#4}{%
    \expandafter\def\csname chronos@#4year\endcsname{#1}%
    \chronos@showdate{#4}%
  },
  chronos date/.style args={#1-#2-#3}{%
    /tikz/chronos set date/.expanded={#1:#2:#3:thing}%
  },
  date format/.code={%
    \chronos@setdateformat{#1}%
  },
}
\makeatother
\begin{document}
  \tikzset{%
    chronos date=2016-04-25,
    date format={date: Y},
    chronos date=2016-04-25,
  }
\end{document}

Again, this works. But it seems a very complicated way to do something very commonly needed i.e. preserving spaces in user input - surely this is not the best approach?!

EDIT 2

OK, this is simpler than EDIT and less hackish than the original. But I'm not sure how safe/robust/proper it is.

This basically replaces the original hackery with

  \tl_replace_all:Nnn \l_chronos_dateformat_tl { ~ } { \c_space_token }

which seems to work for the minimal example, at least. It also seems to work in minimal testing with my full code and seems the 'least weird' of the approaches so far. But that is not to say that it is a good strategy ....

\documentclass[border=10pt,multi,tikz]{standalone}
\usepackage{xparse}
\makeatletter
\ExplSyntaxOn
\tl_new:N \l_chronos_dateformat_tl
\tl_set:Nn \l_chronos_dateformat_tl { d/m/Y }
\cs_generate_variant:Nn \int_abs:n { c }
\cs_new_protected_nopar:Npn \chronos_show_date:n #1
{
  \tl_map_inline:Nn \l_chronos_dateformat_tl
  {
    \str_case:nnF { ##1 }
    {
      { Y } { \int_abs:c { chronos@#1year } } % for each Y in the date format, output the year in YYYY, omitting any leading minus sign
    }
    {
      ##1
    }
  }
}
\cs_new_protected_nopar:Npn \chronos_set_dateformat:n #1
{
  \tl_set:Nn \l_chronos_dateformat_tl { #1 }
  \tl_replace_all:Nnn \l_chronos_dateformat_tl { ~ } { \c_space_token }
}
\NewDocumentCommand \chronos@setdateformat { m }
{
  \chronos_set_dateformat:n { #1 }
}
\NewDocumentCommand \chronos@showdate { o m }
{
  \group_begin:
    \IfValueT { #1 }
    {
      \chronos_set_dateformat:n { #1 }
    }
    \chronos_show_date:n { #2 }
  \group_end:
}
\ExplSyntaxOff
\newcounter{chronos@thingdate}
\tikzset{%
  chronos set date/.code args={#1:#2:#3:#4}{%
    \expandafter\def\csname chronos@#4year\endcsname{#1}%
    \chronos@showdate{#4}% This doesn't usually happen here: instead, the date is saved with a tag such as 'thing' so that the date or parts of the date can be retrieved and formatted later, possibly more than once with different formatting. Multiple tags allow the storing of multiple sets of date/date part information. The information gets used in various ways to calculate, construct and annotate a timeline.
  },
  chronos date/.style args={#1-#2-#3}{%
    /tikz/chronos set date/.expanded={#1:#2:#3:thing}% In the full code 'thing' is one of several tags used to tag dates for the timeline.
  },
  date format/.code={% Let the user change the format of the date without doing anything special to include spaces.
    \chronos@setdateformat{#1}%
  },
}
\makeatother
\begin{document}
  \tikzset{%
    date format={date: Y},
    chronos date=2016-04-25,
  }
\end{document}

Answer 1

There are really two parts to this question, the specifics of the use case and the more general 'how to iterate with spaces' one. I'll tackle the two separately.

---

For the specific case here of replacing numerical part 'holders' with the actual values I would not use a loop at all. There's no particular need to look at every token: a simple search-and-replace will do the job. I'd also not use a single letter token as the 'marker': this is asking for issues. In the following, I've gone for YYYY as the full year, MM as the given month, etc.: one could arrange to have zero-filling, conversion to text and so on.

\documentclass{standalone}
\usepackage{pgfkeys}
\usepackage{expl3}
\ExplSyntaxOn
\tl_new:N \l_chronos_format_tl
\tl_new:N \l__chronos_tmp_tl
\cs_new_protected:Npn \chronos_format:n #1
  { \tl_set:Nn \l_chronos_format_tl {#1} }
\cs_new_protected:Npn \chronos_print:n #1
  {  \__chronos_print:w #1 - - - \q_stop }
\cs_new_protected:Npn \__chronos_print:w #1 - #2 - #3 - #4 \q_stop
  {
    \bool_if:nTF
      {
        \tl_if_blank_p:n {#1} ||
        \tl_if_blank_p:n {#2} ||
        \tl_if_blank_p:n {#3}
      }
      { \ERROR }
      { \chronos_print:nnn {#1} {#2} {#3} }
  }
\cs_new_protected:Npn \chronos_print:nnn #1#2#3
  {
    \tl_set_eq:NN \l__chronos_tmp_tl \l_chronos_format_tl
    \tl_replace_all:Nnn \l__chronos_tmp_tl { YYYY } {#1}
    \tl_replace_all:Nnn \l__chronos_tmp_tl { MM }   {#2}
    \tl_replace_all:Nnn \l__chronos_tmp_tl { DD }   {#3}
    \tl_use:N \l__chronos_tmp_tl
  }
\ExplSyntaxOff
\pgfkeys{
  chronos/.is family,
  chronos/format/.ecode =
    \unexpanded\expandafter{\csname\detokenize{chronos_format:n}\endcsname}%
      {#1},
  chronos/date/.code =
    \unexpanded\expandafter{\csname\detokenize{chronos_print:n}\endcsname}%
      {#1},
}
\begin{document}

\pgfkeys{chronos/format = date: YYYY, chronos/date = 2016-04-25}

\end{document}

I've gone for an approach in which I avoid catcode issues for spaces in \pgfkeys by using expansion to put in the right tokens. One might have a document-level interface for the set up: down to you.

---

If you have more cases than you want to list out as a series of \tl_replace_all:Nnn then I'd probably use a loop over the various cases and a helper for each one

\documentclass{standalone}
\usepackage{pgfkeys}
\usepackage{expl3}
\ExplSyntaxOn
\tl_new:N \l_chronos_format_tl
\tl_new:N \l__chronos_tmp_tl
\cs_new_protected:Npn \chronos_format:n #1
  { \tl_set:Nn \l_chronos_format_tl {#1} }
\cs_new_protected:Npn \chronos_print:n #1
  {  \__chronos_print:w #1 - - - \q_stop }
\cs_new_protected:Npn \__chronos_print:w #1 - #2 - #3 - #4 \q_stop
  {
    \bool_if:nTF
      {
        \tl_if_blank_p:n {#1} ||
        \tl_if_blank_p:n {#2} ||
        \tl_if_blank_p:n {#3}
      }
      { \ERROR }
      { \chronos_print:nnn {#1} {#2} {#3} }
  }
\cs_new_protected:Npn \chronos_print:nnn #1#2#3
  {
    \tl_set_eq:NN \l__chronos_tmp_tl \l_chronos_format_tl
    \clist_map_inline:nn { YYYY , MM , DD }
      {
        \tl_replace_all:Nnx \l__chronos_tmp_tl {##1}
          { \use:c { __chronos_replace_ ##1 :nnn } {#1} {#2} {#3} }
      }
    \tl_use:N \l__chronos_tmp_tl
  }
\cs_new:Npn \__chronos_replace_YYYY:nnn #1#2#3 {#1}
\cs_new:Npn \__chronos_replace_MM:nnn #1#2#3 {#2}
\cs_new:Npn \__chronos_replace_DD:nnn #1#2#3 {#3}
\cs_generate_variant:Nn \tl_replace_all:Nnn { Nnx }
\ExplSyntaxOff
\pgfkeys{
  chronos/.is family,
  chronos/format/.ecode =
    \unexpanded\expandafter{\csname\detokenize{chronos_format:n}\endcsname}%
      {#1},
  chronos/date/.code =
    \unexpanded\expandafter{\csname\detokenize{chronos_print:n}\endcsname}%
      {#1},
}
\begin{document}

\pgfkeys{chronos/format = date: YYYY, chronos/date = 2016-04-25}

\end{document}

---

On the more general issue of looping with spaces, I think that's really a separate question. You rarely need to do such a loop, compared with taking some other more focussed approach. As such, there is not a public function for that type of outcome in expl3 at preset. There is an internal set up to do the job.

You can loop over tokens containing spaces in two ways. The first, if spaces are the only concern, is to use two loops: one that splits on spaces, then one within each 'word'. The latter doesn't have to worry about spaces whilst the outer loop can re-insert them as required. The more sophisticated approach is to also worry about brace groups, which will be lost in the simpler implementation. That can be done either in an expandable or non-expandable manner: only the latter can preserve character tokens for groups. (That is a pretty unusual worry but it's worth noting.) The approach is to use test the 'head' of the input for

• Spaces
• Begin-group tokens
• Anything else

and then use an appropriate auxiliary for each case. See \__tl_act:NNNnn in expl3-code.tex (or in l3tl.dtx) for details of the expl3 version of this idea.

---

The latter approach would look something like

\documentclass{standalone}
\usepackage{pgfkeys}
\usepackage{expl3}
\ExplSyntaxOn
\tl_new:N \l_chronos_format_tl
\tl_new:N \l__chronos_tmp_tl
\cs_new_protected:Npn \chronos_format:n #1
  { \tl_set:Nn \l_chronos_format_tl {#1} }
\cs_new:Npn \chronos_print:n #1
  {  \__chronos_print:w #1 - - - \q_stop }
\cs_new:Npn \__chronos_print:w #1 - #2 - #3 - #4 \q_stop
  {
    \bool_if:nTF
      {
        \tl_if_blank_p:n {#1} ||
        \tl_if_blank_p:n {#2} ||
        \tl_if_blank_p:n {#3}
      }
      { \ERROR }
      { \chronos_print:nnn {#1} {#2} {#3} }
  }
\cs_new:Npn \chronos_print:nnn #1#2#3
  {
    \exp_after:wN \__chronos_loop:wnnn \l_chronos_format_tl
      \q_recursion_tail \q_recursion_stop {#1} {#2} {#3}
  }
\cs_new:Npn \__chronos_loop:wnnn #1 \q_recursion_stop % Three n-type follow
  {
    \tl_if_head_is_N_type:nTF {#1}
      { \__chronos_N_type:Nwnnn }
      {
        \tl_if_head_is_group:nTF {#1}
          { \__chronos_group:nwnnn }
          { \__chronos_space:wwnnn }
      }
    #1 \q_recursion_stop
  }
\cs_new:Npn \__chronos_N_type:Nwnnn #1#2 \q_recursion_stop #3#4#5
  {
    \if_meaning:w \q_recursion_tail #1
      \exp_after:wN \__chronos_end:wnnn
    \fi:
    \str_case:nnF {#1}
      {
        { Y } {#3}
        { M } {#4}
        { D } {#5}
      }
      {#1}
    \__chronos_loop:wnnn #2 \q_recursion_stop {#3} {#4} {#5}
  }
\cs_new:Npn \__chronos_group:nwnnn #1#2 \q_recursion_stop
  {
    {#1}
    \__chronos_loop:wnnn #2 \q_recursion_stop
  }
\exp_last_unbraced:NNo
  \cs_new:Npn \__chronos_space:wwnnn \c_space_tl #1 \q_recursion_stop
  {
    \c_space_tl
    \__chronos_loop:wnnn #1 \q_recursion_stop
  }
\cs_new:Npn \__chronos_end:wnnn #1 \q_recursion_stop #2#3#4 { }
\ExplSyntaxOff
\pgfkeys{
  chronos/.is family,
  chronos/format/.ecode =
    \unexpanded\expandafter{\csname\detokenize{chronos_format:n}\endcsname}%
      {#1},
  chronos/date/.code =
    \unexpanded\expandafter{\csname\detokenize{chronos_print:n}\endcsname}%
      {#1},
}
\begin{document}

\pgfkeys{chronos/format = date: Y, chronos/date = 2016-04-25}

\end{document}

One could write out the tests by hand (for efficiency) but I've stcuk with the pre-built ones in expl3. The core idea here is to use a delimited argument to allow us to grab #1 with a leading space or brace group, and be able to test it. This approach is best used for single-token markers: it is possible though more work to look for multi-token markers this way.

As there is no assignment here I've made the code expandable: this may or may not be desirable.

One might use a lookup table for the token type

\documentclass{standalone}
\usepackage{pgfkeys}
\usepackage{expl3}
\ExplSyntaxOn
\tl_new:N \l_chronos_format_tl
\tl_new:N \l__chronos_tmp_tl
\cs_new_protected:Npn \chronos_format:n #1
  { \tl_set:Nn \l_chronos_format_tl {#1} }
\cs_new:Npn \chronos_print:n #1
  {  \__chronos_print:w #1 - - - \q_stop }
\cs_new:Npn \__chronos_print:w #1 - #2 - #3 - #4 \q_stop
  {
    \bool_if:nTF
      {
        \tl_if_blank_p:n {#1} ||
        \tl_if_blank_p:n {#2} ||
        \tl_if_blank_p:n {#3}
      }
      { \ERROR }
      { \chronos_print:nnn {#1} {#2} {#3} }
  }
\cs_new:Npn \chronos_print:nnn #1#2#3
  {
    \exp_after:wN \__chronos_loop:wnnn \l_chronos_format_tl
      \q_recursion_tail \q_recursion_stop {#1} {#2} {#3}
  }
\cs_new:Npn \__chronos_loop:wnnn #1 \q_recursion_stop % Three n-type follow
  {
    \tl_if_head_is_N_type:nTF {#1}
      { \__chronos_N_type:Nwnnn }
      {
        \tl_if_head_is_group:nTF {#1}
          { \__chronos_group:nwnnn }
          { \__chronos_space:wwnnn }
      }
    #1 \q_recursion_stop
  }
\cs_new:Npn \__chronos_N_type:Nwnnn #1#2 \q_recursion_stop #3#4#5
  {
    \if_meaning:w \q_recursion_tail #1
      \exp_after:wN \__chronos_end:wnnn
    \fi:
    \cs_if_exist_use:cTF { __chronos_replace_ #1 :nnn }
      { {#3} {#4} {#5} }
      {#1}
    \__chronos_loop:wnnn #2 \q_recursion_stop {#3} {#4} {#5}
  }
\cs_new:Npn \__chronos_replace_Y:nnn #1#2#3 {#1}
\cs_new:Npn \__chronos_replace_M:nnn #1#2#3 {#2}
\cs_new:Npn \__chronos_replace_D:nnn #1#2#3 {#3}
\cs_new:Npn \__chronos_group:nwnnn #1#2 \q_recursion_stop
  {
    {#1}
    \__chronos_loop:wnnn #2 \q_recursion_stop
  }
\exp_last_unbraced:NNo
  \cs_new:Npn \__chronos_space:wwnnn \c_space_tl #1 \q_recursion_stop
  {
    \c_space_tl
    \__chronos_loop:wnnn #1 \q_recursion_stop
  }
\cs_new:Npn \__chronos_end:wnnn #1 \q_recursion_stop #2#3#4 { }
\ExplSyntaxOff
\pgfkeys{
  chronos/.is family,
  chronos/format/.ecode =
    \unexpanded\expandafter{\csname\detokenize{chronos_format:n}\endcsname}%
      {#1},
  chronos/date/.code =
    \unexpanded\expandafter{\csname\detokenize{chronos_print:n}\endcsname}%
      {#1},
}
\begin{document}

\pgfkeys{chronos/format = date: Y, chronos/date = 2016-04-25}

\end{document}

(The code we have for this type of loop in expl3 is internal as we've not wanted to promote it more generally. If there are more use cases I'll look to provide a documented code interface.)