# About catcodes and Family in LaTeX3/expl3

This is not so much the display of an issue or problem I'm facing out of my wish to use as much LaTeX3/expl3 as possible in my TeX-based work, as a request for advice on the proper ways in that scenario.

What's the right way (or the right ways) to change catcodes in LaTeX3/expl3? In particular, when using l3keys to set some token list and retrieve later on its content making the "space char" regain catcode 10 (the regular way it is outside the expl3 block codes), what's the suggested way to proceed? Could somebody provide a more extended description of the way that \tl_set_rescan:Nnn is to be used? I'm thinking in particular of the setup argument.

What follows is a possible context where this shows up: I'd like to be able to retain the spaces in the argument to the \my_name:n function without using ~.

\RequirePackage{expl3}
\documentclass{memoir}

\ExplSyntaxOn

\tl_new:c { g_my_name_tl }

\cs_new:cpn { my_name :n } #1
{
\tl_gset:cn { g_my_name_tl } {#1} %{ \tl_to_str:n {#1} }
}

\my_name:n {the name}

\ExplSyntaxOff

\begin{document}
\g_my_name_tl
\end{document}

• Ideally you should never change catcodes in any command to be used in the document, it's a method of last resort. If the token list has come from the document it will normally have normal spaces even if you write the code to handle it in an expl3 syntax area. Perhaps you could give an example code of what you are wanting to do – David Carlisle Sep 7 '13 at 10:39
• @DavidCarlisle: My only actual need for this so far has been avoiding to use the ~ character when setting some key defined in terms of a "token list property". This is something I do in the preamble, and the expl3 syntax is still "switched on". But I might need to do more than that at some point in time, and I find the interface3 doc too sketchy on the use of the "rescan" function. – Marcos Sep 7 '13 at 10:53
• rescan is is more or less just etex \scantokens which is a tricky beast to tame. It is best not to use it unless you are in some really custom parsing situation. – David Carlisle Sep 7 '13 at 11:06
• I should note that if you enter a b c in expl3 conventions it is not possible to rescan to get "normal" spaces. Spaces in expl3 code are ignored no record of them is left in the token list constructed. The space character is seen but no token is produced by TeX's parser when it sees a character with catcode set to ignore. a b c makes the three tokens abc so rescanning after changing the catcode of space will give abc. – David Carlisle Sep 7 '13 at 11:09

In general, we strongly discourage altering category codes. Indeed, I'd like to imagine that longer-term we can provide a sufficient set of tools that catcode changes are really no needed at all (for example, my feeling is we should have a larger but fixed set of 'active' characters at the document level).

In terms of spaces, within code blocks spaces () are ignored while the tilde (~) can be used to represent a space. If you want a 'hard' space, the 'document-level' approach is probably not quite correct anyway, but at present we don't have a version of the command, so I'd use \nobreakspace.

As mentioned in comments, if you are grabbing document-level stuff then it will be tokenized with as a normal space, so you should have no issues. What I would do in a keys context is not use spaces in key names: there is a reason all of the ones in for example siunitx use hyphens!

Using \tl_set_rescan:Nnn is tricky at best, and I'd strongly discourage it. Indeed, I am considering removing it entirely for being used in siunitx as I think my earlier choice of approach is wrong in this regard. Certainly I would want to see a definite example before recommending its use.

Taking the specific example in the question, you are really using a 'code level' function in the wrong way. There are really two cases. The first is where it makes sense to set up some 'document like' material in the code, for example

\keys_define:nn { mymodule }
{
key .tl_set:N  = \l_my_tl ,
key .initial:n = { Some~text }
}


in which case you have to use 'code like' input (~ for spaces). This will almost always be used with short pieces of text.

(In this context, it's worth noting that we did at one point do 'catcode trickery' for message text, as it is a little tedious typing in all of those ~ in messages. However, when we looked at things again it was clear this was not a good plan.)

The second case is where you really are looking at proper document-like input, in which case use a document command

\cs_new_protected:Npn \my_command:n #1
{
...
}
\NewDocumentCommand { \mycommand } { m }
{ \my_command:n {#1} }


In particular, remember that while you can use expl3 code in a document preamble, the main way it should be used is in packages, with the resulting code then exploited in new document commands via \usepackage.

• My original aim was just to avoid the need to type the ~ character when setting a key collecting raw data to be used to typeset a header. In my post about using a stack structure I've hinted at how I was trying to do this, namely, through the function \tl_to_str:(N|n|...). In that setting, I was using functions instead of keys to set token lists, though, and this approach doesn't seem to work in the functions I was setting there. I wonder what would be the right way to proceed. – Marcos Sep 7 '13 at 15:51
• Very insightful on what belongs where. It is something of an irony that where I now have expl3 functions, I used to have xparse \NewDocumentCommands. My problem was how to automate the definition of the latter in terms of a list as I do for the expl3 functions on this post that you're already familiar with ;-) Should I issue a new post on that topic? – Marcos Sep 7 '13 at 19:28
• @Marcos The team are actually pretty wary of any 'auto-generation' of document commands. What you have in the example seems to be a re-implementation of \author, for example. Also note that the reason you almost always need expl3 and xparse is precisely because we are trying to have a proper 'code' and 'document' separation. – Joseph Wright Sep 7 '13 at 19:31
• So, is it considered a "better practice" to define a number of commands on a one-to-one basis than automating the process even though they are different from each other only in terms of a short substring? – Marcos Sep 7 '13 at 19:45