# Storing unexpanded macro definitions (including #1, #2, …) for later use (now includes full usecase)

## Preliminary Info

I'm working on a new LaTeX package to make scoping more flexible. To do this I'll be capturing and reintroducing user defined commands behind the scenes.

(For a comprehensive description of this package, skip to the Motivation section.)

• For storage and retrieval, I'm (ab)using the excellent datatool package.
• This code won't show the scoping stuff. I'm reducing it to the issue at hand.

## What I Have So Far

\usepackage{xparse}
\usepackage{datatool}
\usepackage{etextools}

\makeatletter

\newcommand{\sc@db@name}{scopedcommands}

\DTLnewdb{\sc@db@name}

\newcommand{\sc@db@assignlist}{
\sc@db@col@index      =index,
\sc@db@col@star       =star,
\sc@db@col@command    =command,
\sc@db@col@arguments  =arguments,
\sc@db@col@default    =default,
\sc@db@col@definition =definition%
}

\newcounter{sc@defcount}

\DeclareDocumentCommand{\snewcommand}{s+mo+o+m}{%
\DTLnewrow{\sc@db@name}%
\dtlexpandnewvalue%
\DTLnewdbentry{\sc@db@name}{index}    {\thesc@defcount}%
\DTLnewdbentry{\sc@db@name}{star}     {\IfBooleanT{#1}{*}}%
\DTLnewdbentry{\sc@db@name}{command}  {\noexpand#2}%
\DTLnewdbentry{\sc@db@name}{arguments}{\IfNoValueF{#3}{[#3]}}%
\dtlnoexpandnewvalue%
\IfValueTF{#4}{\DTLnewdbentry{\sc@db@name}{default}{[#4]}}%
{\DTLnewdbentry{\sc@db@name}{default}{}}%
\DTLnewdbentry{\sc@db@name}{definition}{#5}%
}

\newcommand{\definestoredcommand}[1]{%
\expandnext{\DTLassign{\sc@db@name}{#1}}{\sc@db@assignlist}%
%
\expandnext{%
\expandaftercmds{%
\expandaftercmds{%
\expandnext{%
\expandaftercmds{%
\newcommand%
}{%
\sc@db@col@star%
}}%
{\sc@db@col@command}%
}{%
\sc@db@col@arguments%
}}{%
\sc@db@col@default%
}}%
{\sc@db@col@definition}
}

\makeatother


It works quite well for commands without parameters:

\def\testfoo{foo}
\snewcommand*{\foo}{\testfoo}     % index 1
\snewcommand*{\foo}{\testfoo bar} % index 2

\definestoredcommand{1}
\foo % foo

\let\foo\relax
\definestoredcommand{2}
\foo % foobar


## Problem

But if I add parameters and actually use them in the command definition, I get errors from the deepest bowels of datatool: "Illegal parameter number in definition of \@dtl@tmp."

Obviously I should be escaping the hash character in the command definition.

## Questions

1. How can I escape any possible hash characters so they'll be dormantly stored in the database, and then unescape them so they'll do their job?
2. Is there anything else that might still go wrong with this?

## Motivation (new info)

I should show you what the package is supposed to do when it's finished. It's something I need for writing my PhD thesis, but I think it may be useful for other TeXers.

### Basics

In its simplest form it would allow you to open a named scope multiple times and have the referencing environment continue from one to the other, but not in between. The mandatory argument holds the name:

\begin{scope}{s1}
\newcommand{\foo}{FOO}
\foo   % FOO
\end{scope}

\foo   % <error: undefined>

\begin{scope}{s1}
\foo   % FOO
\end{scope}


(Everything up until this point already works.)

Of course, nested scopes behave as expected. But scopes can also behave as though they are nested while being textually separate, and therein lies the real power of this package. You do this by specifying a comma-separated list of superscopes in the optional argument:

\begin{scope}{s1}
\newcommand{\foo}{FOO}

\begin{scope}{s1sub}
\newcommand{\baz}{BAZ}
\end{scope}

\baz   % <error: undefined>
\end{scope}

\begin{scope}[s1sub]{s2}
\newcommand{\bar}{\foo BAR}
\bar   % FOOBAR
\baz   % BAZ
\end{scope}

\begin{scope}{s1}
\bar   % <error: undefined>
\baz   % <error: undefined>
\end{scope}


The package would also allow you to do something else you normally can't do with scopes: have a partial nesting hierarchy, i.e. inherit the commands from two or more unrelated scopes. Some preemptive reassurement: a single-origin command inherited via multiple paths will cause no trouble:

\begin{scope}{s1}
\newcommand{\foo}{FOO}
\end{scope}

\begin{scope}[s1]{s2}
\newcommand{\bar}{\foo BAR}
\end{scope}

\begin{scope}[s1]{s3}
\newcommand{\baz}{\foo BAZ}
\bar   % <error: undefined>
\end{scope}

\begin{scope}[s2,s3]{s4}
\foo   % FOO   <no problem>
\bar   % FOOBAR
\baz   % FOOBAZ
\end{scope}


Last but not least, when you 'join' two scopes in this way, you have full control over how to resolve any conflicts between their referencing environments:

\begin{scope}{s1}
\newcommand{\foobar}{FOO}
\end{scope}

\begin{scope}{s2}
\newcommand{\foobar}{BAR}
\end{scope}

\begin{scope}[s1,s2]{s3}
% <no problem so far>
\foobar   % <error: ambiguous>
\renewcommand{\foobar}{\from{s1}\foobar \from{s2}\foobar}
\foobar   % FOOBAR
\end{scope}


### Extras

The nice thing is, you could optionally get rid of all the overhead by embedding scopes in existing constructs. For my intended use-case:

• the document environment forms a root scope
• every chapter/section/etc. forms a properly nested scope with a name derived from their \label
• \label takes an optional argument specifying superscopes

### My Usecase

My PhD thesis will consist of a formal framework (delta modeling: find related publications here), introducing new definitions/theorems step-by-step, building on earlier ones. The thesis has a partial recommended reading-order. Some independent concepts build on the first chapter. Later chapters may then combine these concepts. Building on this package, I can ensure that:

• References to definitions/theorems/notation only occur later in the partial reading order than where they originated,
• newly introduced names and notation are unambiguous within their branch,
• notation is always consistent and
• notation can be automatically extracted and put in
• sentence fragments, e.g., "for any deltas $x, y \in D$" and
• a glossary which reflects the reading order.

Example:

\chapter{Identity Function} \label{identity}

\newcommand*{\i}{I}

$\i$ is a function such that $\i(x) = x$.

\chapter{Derivatives} \label{derivatives}

\newcommand*{\d}[1]{#1'}

Given function $f$, $\d{f}$ is the derivative of $f$.

% no clue about \i here

\chapter{Boring Conclusion} \label[identity,derivatives]{boring}

Taking the information from chapters \ref{identity}
and \ref{derivatives}, we can now see that:

$\forall x \in \mathbb{R}: \d{\i}(x) = 1$


As you see, it's mostly the partial order I'm interested in. But I'm choosing to make the package more general for two reasons:

• It may be useful to others, maybe for purposes I haven't yet come up with and
• the partially ordered scopes which transform the referencing environment can be seen as deltas in a delta model. In other words, it's interesting for my PhD research. A description of this package will actually be part of my thesis, formalized within the framework. I love self-reference.
• The best way to store command definitions is to use command definitions; it's not clear what you want to achieve from the complicated example and the rather scanty example. – egreg Oct 16 '12 at 17:42
• egreg: Hehe. I do feel the fool, always over-complicating things. :-) Your comment actually helped me. (Put it in answer form and I'll accept it.) But I'm still going to have to keep track of those commands, so the eventual code will still look quite similar. ----- You're also right that I should motivate questions like this with a clear example of what my package is eventually supposed to do. Can I put something like that on TeXexchange even if it's not a question by itself? ----- Thanks! – mhelvens Oct 16 '12 at 18:18
• I'd answer if I understood the code. Please, add some motivation and an example of how it should work. You surely can edit and improve your question. – egreg Oct 16 '12 at 19:41
• In particular please always make code sections complete documents, I tried to make an example from your question but just got ! Undefined control sequence.<argument> \DTLassign  which presumably means there is another package to load but it is very hard to guess given the fragments in the question – David Carlisle Oct 16 '12 at 21:18
• No, the code is complete (just add \begin{document}, etc.). But you need version v2.11 of datatool. And note that I have created a fully working version based on egreg's inadvertent tip. I'll post it as an answer here tomorrow(ish). – mhelvens Oct 16 '12 at 21:23

In his comment above, egreg showed me I was over-complicating the issue. I now solved the problem by storing each new definition in a uniquely named global macro. The database now only maps a user-defined <scope-name, macro-name> pair to the name of the global macro containing the appropriate definition. Here's the code I have so far. It includes some of the scoping stuff from my motivation above:

\documentclass{article}

\usepackage{xparse}
\usepackage{datatool}
\usepackage{etextools}
\usepackage{letltxmacro}

\makeatletter

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Global versions of \newcommand, etc. I needed this anyway, might as well
% make it complete.

\def\gc@common{%
{%
\LetLtxMacro{\gc@temp}{\relax}%
\begingroup\edef\x{\endgroup%
\noexpand\newcommand%
\IfBooleanT{##1}{*}%
{\noexpand\gc@temp}%
\IfNoValueF{##3}{[##3]}%
\IfNoValueF{##4}{[\unexpanded{##4}]}%
{\unexpanded{##5}}%
}\x%
\GlobalLetLtxMacro{##2}{\gc@temp}%
}%
}

% TODO: Yeah, they do the same thing now. Fix their behavior.
\expandnext{\DeclareDocumentCommand{\gensurecommand} {s+mo+o+m}}{\gc@common}
\expandnext{\DeclareDocumentCommand{\gnewcommand}    {s+mo+o+m}}{\gc@common}
\expandnext{\DeclareDocumentCommand{\grenewcommand}  {s+mo+o+m}}{\gc@common}
\expandnext{\DeclareDocumentCommand{\gprovidecommand}{s+mo+o+m}}{\gc@common}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newcommand{\sc@currentscope}{}

\DTLnewdb{sc@db}

\newcommand{\sc@db@assignlist}{      %
\sc@db@col@index      =index,    %
\sc@db@col@scope      =scope,    %
\sc@db@col@command    =command,  %
\sc@db@col@definition =definition%
}

\newcommand{\sc@defit}{%
\expandnexttwo\LetLtxMacro{\sc@db@col@command}{\sc@db@col@definition}%
}

\newcounter{sc@count}

\def\sc@common{%
\DTLnewrow{sc@db}%
\def\sc@def{\csname sc@def@\thesc@count\endcsname}%
\dtlexpandnewvalue%
\DTLnewdbentry{sc@db}{index}     {\thesc@count}%
\DTLnewdbentry{sc@db}{scope}     {\sc@currentscope}%
\DTLnewdbentry{sc@db}{command}   {\noexpand##2}%
\DTLnewdbentry{sc@db}{definition}{\sc@def}%
\begingroup\edef\x{\endgroup%
\noexpand\gnewcommand%
\IfBooleanT{##1}{*}%
{\sc@def}%
\IfNoValueF{##3}{[##3]}%
\IfNoValueF{##4}{[\unexpanded{##4}]}%
{\unexpanded{##5}}%
}\x%
\expandnext{\DTLassign{sc@db}{\thesc@count}}{\sc@db@assignlist}%
}

% TODO: Yeah, most of these do the same thing now. Fix their behavior.
\expandnext{\DeclareDocumentCommand{\sensurecommand}{s+mo+o+m}}%
{\sc@common\sc@defit}
\expandnext{\DeclareDocumentCommand{\snewcommand}{s+mo+o+m}}%
{\sc@common%
\expandnext\newcommand{\sc@db@col@command}{}%
\sc@defit}
\expandnext{\DeclareDocumentCommand{\srenewcommand}{s+mo+o+m}}%
{\sc@common\sc@defit}
\expandnext{\DeclareDocumentCommand{\sprovidecommand}{s+mo+o+m}}%
{\sc@common\sc@defit}

\newcommand{\sc@forall}[2][\boolean{true}]{%
\expandnext{\DTLforeach[#1]{sc@db}}{\sc@db@assignlist}{#2}%
}

\newenvironment{scope}[1]{%
\renewcommand{\sc@currentscope}{#1}%
\sc@forall[\DTLiseq{\sc@db@col@scope}{\sc@currentscope}]{\sc@defit}%
}{}

\newcommand{\from}[2]{%
{% explicit local scope
\sc@forall[\DTLiseq{\sc@db@col@scope}{#1}]{\sc@defit}%
#2%
}%
}

\newcommand{\scshow}[1][\boolean{true}]{%
\begin{tabular}{|r|l|l|l|}%
\hline%
& \bfseries scope & \bfseries command & \bfseries definition%
\sc@forall[#1]{%
\\\hline        %
\sc@db@col@index                                     &%
\ttfamily\expandnext\detokenize{\sc@db@col@scope}    &%
\ttfamily\expandnext\detokenize{\sc@db@col@command}  &%
\ttfamily\expandnext\detokenize{\sc@db@col@definition}%
}%
\\\hline%
\end{tabular}%
}

\makeatother

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}                                                               %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\let\bar\relax

\begin{scope}{main}
\snewcommand*{\foo}[1]{foo(#1)}
\snewcommand*{\bar}{bar}
\foo{\bar}   % foo(bar)
\end{scope}

\begin{scope}{hidden}
\snewcommand{\secret}[1]{(secret #1)}
\end{scope}

\begin{scope}{main}
\foo{\bar\bar}   % foo(barbar)

\from{hidden}{\secret{discovered}}   % (secret discovered)
\end{scope}

\scshow   % <prints handy table showing the database>

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}                                                                 %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


Incomplete, of course. But it solves the problem from my original question. Try it out! (Note that it requires version v2.11 of datatool.) Comments would be appreciated!

• Just curious, did you ever submit this to CTAN? Several years ago I ran into an issue where I thought it would be useful to have more flexibility with scopes. But, I don't recall exactly what the issue was though. – Peter Grill May 18 '18 at 7:27