Due to the 30000-character-limit for answers I needed to split my answer in two parts.
This is part 1 of my answer.
Part 1 of my answer holds general explanations of the workings of TeX and of the code/working example provided in part 2.
Part 2 of my answer holds the working example.
You use the phrase "raw content" without clarifying what it shall mean/denote.
I can offer a generic interface \UD@GatherCodeSnippetLoop
which switches to verbatim-catcode-régime (category codes of special characters ⟨space-character⟩
, \
, {
, }
, $
, &
, #
, ^
, _
, %
, ~
switched to 12(other)) and character by character reads from the .tex-input-file and tokenizes and accumulates things until encountering a character-sequence \end{⟨expansion of \@currenvir⟩}
, applying \string
to every accumulated non-space-character for transforming everything, e.g., characters usually active due to the package inputenc, to catcode 12(other). (Applying \string
to non-space-character-tokens only is for the following reason: Space-characters already have category-code 12 due to the switching to verbatim-catcode-régime. Applying \string
would transform them into explicit space-tokens of category code 10(space) as spaces/space-tokens are the only exception with \string
: \string
transforms everything but spaces to category code 12(other) and transforms every space to category code 10(space). )
\UD@GatherCodeSnippetLoop
can be used for defining different environments with differing behavior, having in common that the body of the environment is gathered from the .tex-input-file as a sequence of explicit character tokens of category code 12(other).
Such environments can be used for storing these sequences of explicit character tokens of category code 12(other) as macros.
After expanding these macros, processing the sequences of explicit character tokens of category code 12(other) by \scantokens
yields the same as processing the characters forming bodies of the environments via TeX's eyes/mouth/gullet/stomach etc directly without the actions of switching catcode-régime and saving things away—unless the catcode-régime in effect at the time of carrying out such an environment differs from the catcode-régime in effect at the time of carrying out the macro/at the time of \scantokens
' retokenization of the single characters that formed the body of the environment.
In the example in part 2 the generic interface \UD@GatherCodeSnippetLoop
is defined and environments DefineCodeSnippet
, RedefineCodeSnippet
, PrependToCodeSnippet
, AppendToCodeSnippet
based on \UD@GatherCodeSnippetLoop
are defined. Additionally user-level-macros
\ConcatCodeSnippets
, \ExecuteCodeSnippet
, \DeliverCodeSnippetToMacro
, \DeliverCodeSnippetToTokens
are defined. All the environments and user-level-macros also process some argument/arguments denoting IDs of code-snippets.
\begin{DefineCodeSnippet}{⟨ID⟩}
Text Text Text
Text
\end{DefineCodeSnippet}
reads the body of the environment from the .tex-input-file as a sequence of explicit catcode-12(other)-character-tokens and defines the macro \CodeSnippet⟨ID⟩
to yield this sequence of catcode-12(other)-character-tokens by toplevel-expansion.
\begin{RedefineCodeSnippet}{⟨ID⟩}
Text Text Text
Text
\end{RedefineCodeSnippet}
redefines \CodeSnippet⟨ID⟩
accordingly.
\begin{PrependToCodeSnippet}{⟨ID⟩}
Text Text Text
Text
\end{PrependToCodeSnippet}
prepends lines to \CodeSnippet⟨ID⟩
.
\begin{AppendToCodeSnippet}{⟨ID⟩}
Text Text Text
Text
\end{AppendToCodeSnippet}
appends lines to \CodeSnippet⟨ID⟩
.
The macro
\ConcatCodeSnippets{⟨ID of source snippet 1⟩}{⟨ID of source snippet 2⟩}{⟨target ID⟩}
defines \CodeSnippet⟨target ID⟩
to yield:
⟨sequence of explicit catcode-12(other)-character-tokens denoted by ID of source snippet 1⟩
⟨explicit ⟨carriage-return⟩-character-token of category code 12 (other)⟩
⟨sequence of explicit catcode-12(other)-character-tokens denoted by ID of source snippet 2⟩
The starred variant \ConcatCodeSnippets*
allows to override/redefine \CodeSnippet⟨target ID⟩
.
The macro
\ExecuteCodeSnippet{⟨ID⟩}
does
\scantokens\expandafter{\CodeSnippet⟨ID⟩%}
, the trailing %
being of catcode 12(other) as well so that it will be written by \scantokens
' fake-writing without problems and be recognized by \scantokens
' reading-back under its usual catcode 14(comment) whereby it then prevents the coming into being of a spurious token due to TeX's insertion of the endline-character.
Before passing things to \scantokens
⟨carriage-return⟩-characters are replaced by ⟨line-feed⟩-characters.
In order to understand the reason for the latter you need to know about some details of how TeX works:
At the time when TeX reads a line of a .tex-input-file and pre-processes it, right before tokenization, a character is appended whose code-point-number in TeX's internal character-encoding-scheme (which with traditional TeX is ASCII and with LuaTeX/XeTeX is unicode) equals the value of the integer-parameter \endlinechar
.
Usually the value of \endlinechar
is 13 while code-point number 13 denotes the ⟨carriage-return⟩-character both in ASCII and in Unicode.
So usually a ⟨carriage-return⟩-character is appended while reading/preprocessing a line of .tex-input.
In TeX's ^^
-notation the ⟨carriage-return⟩-character is also denotable as ^^M
as M is the 13th letter in the uppercase-alphabet.
At the time when TeX writes explicit character-tokens unexpanded to external text file or screen, TeX with each explicit character-token checks whether its character-code (=the number of the code-point of the character in question in TeX's internal character-encoding-scheme) equals the value of the integer-parameter \newlinechar
.
If so, then the corresponding character is not written but the character-token in question is taken for a signal to create in the external text file or on the screen a transition from the current line to a next line.
The result of this is platform-dependent. In text-files, depending on the platform, the ⟨carriage-return⟩-character or the ⟨line-feed⟩-character or a combination of both is used for denoting the transition from one line of a text to the next line of text.
Usually the value of \newlinechar
is 10 while code-point number 10 denotes the ⟨line-feed⟩-character both in ASCII and in Unicode.
So at the time of writing things usually ⟨line-feed⟩-character-tokens are signals for creating a transition from one line of a text-file to the next line of that text-file.
In TeX's ^^
-notation the ⟨line-feed⟩-character is also denotable as ^^J
as J is the 10th letter in the uppercase-alphabet.
\scantokens
processes a set of already tokenized tokens as follows:
First TeX fakes unexpanded-writing these tokens to external text file.
Then TeX uses that faked external file as souce of .tex-input (as if you had applied the \input
-primitive for switching to another source of .tex-input) and reads/preprocesses/tokenizes/digests it outgoing from the catcode-régime which is current at the time of carrying out \scantokens
.
In our situation the set of already tokenized tokens was tokenized under verbatim-catcode-régime, so that only (non-special) explicit character-tokens came into being, and each character-token was transformed to have category-code 12(other). This implies that the ⟨carriage-return⟩-characters, which—due to the above-mentioned \endlinechar
-mechanism—were inserted at the ends of lines during the stage of pre-processing, got tokenized as (non-special) explicit character-tokens of category code 12(other) and character code 13.
Usually TeX writes such tokens to file in ^^
-notation, i.e., instead of writing some combination of ⟨carriage-return⟩-characters and/or ⟨line-feed⟩-characters or whatever, the character-sequence ^
, ^
, M
gets written, and characters written subsequently also end up in the same line.
If you want \scantokens
' fake-writing to instead create a transition from one line to the next, you can either assign \newlinechar
the value 13 right before applying \scantokens
, or, remembering that the value of the \newlinechar
-parameter usually denotes the ⟨line-feed⟩-character (10/^^J
), you can have TeX replace each explicit catcode-12(other)-⟨carriage-return⟩-character-token by an explicit catcode-12(other)-⟨line-feed⟩-character-token before applying \scantokens
.
I prefer the latter because it does not require changes of whatsoever parameters of TeX being in effect while \scantokens
is carried out, in edge cases probably leading to behaviour deviating from what is considered the "usual thing".
( If you would let the fake-writing of \scantokens
just write the sequence ^
, ^
, M
instead of having the fake-writing-routine create a transition from one line to the next, inputting/reading/tokenizing/digesting/processing the fake-file under normal catcode-régime would be a problem: ^
, ^
, M
would be taken for ^^M
, i.e., for the ^^
-notation of the ⟨carriage-return⟩-character. Under normal catcode-régime the ⟨carriage-return⟩-character has category code 5(end of line) which means that it causes TeX
(!!!) to cease processing the line, which imples that every character written in the same line behind the character-sequence ^
, ^
, M
by \santokens
' fake-writing would be lost(!!!), and
depending on the current state of the reading apparatus, to append to the token-stream that goes down TeX's gullet, either no token at all (state S), or an explicit space-token (state M), or the control-word-token \par
(state N).
)
The macro
\DeliverCodeSnippetToMacro{⟨ID⟩}{⟨tokens⟩}
delivers
⟨tokens⟩{⟨expansion of \CodeSnippet⟨ID⟩⟩}
The macro
\DeliverCodeSnippetToTokens{⟨ID⟩}{⟨tokens⟩}
delivers
⟨tokens⟩⟨expansion of \CodeSnippet⟨ID⟩⟩
( The difference between \DeliverCodeSnippetToMacro
and \DeliverCodeSnippetToTokens
is that the further delivers ⟨expansion of \CodeSnippet⟨ID⟩⟩
nested between a pair of curly braces of catcode 1(begin group)/catcode 2(end group) while the latter delivers things without such a brace-pair. )
With the enviroments one leading ⟨carriage-return⟩-character (if present) and one trailing ⟨carriage-return⟩-character (if present) will be removed from the ⟨sequence of catcode-12-characters formed by the body of the environment⟩ before (re)defining \CodeSnippet⟨ID⟩
.
This way
\begin{DefineCodeSnippet}{foobar}
Text Text Text
Text
\end{DefineCodeSnippet}
yields the same as
\begin{DefineCodeSnippet}{foobar}Text Text Text
Text\end{DefineCodeSnippet}
, namely:
\newcommand*\CodeSnippetfoobar{Text Text Text⟨explicit ⟨carriage-return⟩-character-token of catcode 12(other)⟩Text}
; every token of \CodeSnippetfoobar
's ⟨definition-text⟩ being an explicit character-token of category-code 12(other).
In the edge case of the body of the environment consisting of ⟨carriage-return⟩-characters only, only one of them is removed as in this case it is assumed that with n ⟨carriage-return⟩-characters one wishes to denote (n-1) empty lines. (The first ⟨carriage-return⟩-character denotes the transition from the line holding \begin{⟨environment⟩}
to the first line holding "emptiness".)
Be aware that TeX's pre-processing of lines of .tex-input in any case removes sequences of ⟨space⟩-characters that occur right at the ends of lines of .tex-input.
Therefore you cannot preserve sequences of ⟨space⟩-characters that occur right at the ends of lines of .tex-input for re-processing by TeX (e.g. via \scantokens
).
The environments DefineCodeSnippet
etc rely on temporarily changing the category-code-régime and obtaining the tokens forming their body
by reading/tokenizing content from .tex-input files while the changed category-code-régime is in effect.
Therefore the environments DefineCodeSnippet
etc cannot be used inside macro-definitions/macro-arguments and the like where stuff gets tokenized already at the time of gathering the tokens that form the macro's definition/at the time of gathering the tokens that form the macro's argument(s).
b
argument of xparse and storing in a macro. They have have slight differences, but your description doesn't allow to decide which you need.multilines
environment content you wish to use again and again? If so,\def\z{% A B C } \begin{multilines} \z \end{multilines} \expandafter\calctree\expandafter{\z} \expandafter\calctree\expandafter{\z}
multilines
in context with\calctree
. Please, explain.