# How to expand values stored in a token defined by newtoks

In this project, the directory structure and file names are based on three parameters, with there being one exception to both.

## Directory Path: \GetTexFilePath{}{}{}

The directory path is based on the top level directory (#1), the first parameter passed to the macro.

Normally the path is of form ../#1/xxx#2/#3/, except when #1=\SpecialCase in which case, the xxx is not part of the path.

## File Names: \GetTexFileName{}{}{}

The file name is dependent on the current value of the token: \the\CurrentTokenValue.

The files names are of the form #1-#2-#3, except for one special case when \the\CurrentTokenValue=\SpecialCase, in which case, the file name is just #3.

Below I am using the expl3 solution adapted from Text being output when none should be.

As requested by @egreg, I have added a section at the beginning to test the various cases and have added the expected output in the comments. Have also added red color to the failing output, and blue color to the correct output.

The first two sections following the above list, work just fine, but the third case where the file name depends on the value of \the\CurrentTokenValue fails.

Also note that in section 3, #1=\the\CurrentTokenValue and the directory name correctly does not have the xxx in it (which is correct). But the file name did not get shortened. Section 3 yields:

../special/02/z/special-02-z

but the correct value for Case 3 should be

../special/02/z/z

## Questions:

1. How do I adjust \GetTexFileName so that the file name depends on the token?
2. Why did this expansion issue of tokens not have a problem when used in the file path, just the file name?

I need to use the concatenated output of the path and file name macros (with a / in between) to be able to to check \IfFileExists.

• I have confirmed that changing the \def\SpecialValue{special} to \protected\def\SpecialValue{special} as per @egreg's answer works for all these cases. My concern with that is I don't know if that will have any other effect on the usage of \SpecialValue, and hence a solution local to \GetTexFilePath and \GetTexFileName would be preferable.
• Bruno's suggestion to add \expandafter\empty before the first parameter to \str_if_eq:xxTF works for this test cast, but fails in my actual usage for some reason.

## Notes:

• The complaint about the file not being found is not the problem.
• All working cases are in blue.
• All failing cases are in red.

## Code:

\documentclass{article}
\usepackage{xcolor}
\usepackage{xparse}

\def\SpecialValue{special}
\newtoks{\CurrentTokenValue}

\ExplSyntaxOn
% Determine path: Always #1/xxx#2/#3 unless #1=\SpecialValue
\NewDocumentCommand{\GetTexFilePath}{m m m}{%
\str_if_eq:xxTF{#1}{\SpecialValue}{../#1/#2/#3}{../#1/xxx#2/#3}%
}%

% Determine file name: Always #1-#2-#3 unless \the\CurrentTokenValue=\SpecialValue.
\NewDocumentCommand{\GetTexFileName}{m m m}{%
\str_if_eq:xxTF{\the\CurrentTokenValue}{\SpecialValue}{#3}{#1-#2-#3}%
}%
\ExplSyntaxOff

\newcommand*{\GetFileNameWithPath}[3]{%
\GetTexFilePath{#1}{#2}{#3}/%
\GetTexFileName{#1}{#2}{#3}%
}%

\newcommand*{\TestFile}[2][red]{%
\noindent%
Token Value = \the\CurrentTokenValue%
\par\noindent%
\IfFileExists{#2}{%
\textcolor{#1}{File \FileWithPath\ found.}%
}{%
}%
}%
\begin{document}
\CurrentTokenValue={NonSpecial}%

\color{blue}% These work (up until the change to red color)
1a. \GetTexFilePath{A}{B}{C}             \par%          ../A/xxxB/C           Note: xxx
2a. \GetTexFilePath{\SpecialValue}{B}{C} \par%          ../special/B/C        Note: no xxx
3a. \GetTexFilePath{\the\CurrentTokenValue}{B}{C} \par% ../NonSpecial/xxxB/C  Note: xxx
%   ----------------
4a. \GetTexFileName{A}{B}{C}             \par%          A-B-C
5a. \GetTexFileName{\SpecialValue}{B}{C} \par%          special-B-C
6a. \GetTexFileName{\the\CurrentTokenValue}{B}{C} \par% NonSpecial-B-C

\smallskip\hrule\smallskip

% Changing the value of \CurrentTokenValue to this one special value,
% should only change the file name, and should not effect the
% file path.  Absolutely no reason why the first two here should be
% any different than above, but included here for completeness.
\CurrentTokenValue={\SpecialValue}%

1b. \GetTexFilePath{A}{B}{C}             \par%          ../A/xxxB/C           Note: xxx
2b. \GetTexFilePath{\SpecialValue}{B}{C} \par%          ../special/B/C        Note: no xxx
\color{red}% These next 4 fail
3b. \GetTexFilePath{\the\CurrentTokenValue}{B}{C} \par% ../special/B/C        ** FAILS **
%   ---------------
4b. \GetTexFileName{A}{B}{C}             \par%          C                     ** FAILS **
5b. \GetTexFileName{\SpecialValue}{B}{C} \par%          C                     ** FAILS **
6b. \GetTexFileName{\the\CurrentTokenValue}{B}{C} \par% C                     ** FAILS **
\color{black}

\bigskip\hrule\hrule\bigskip

\CurrentTokenValue={NonSpecial}

\section{Normal Case: Works}
Neither the first parameter, nor the \verb|\the\CurrentTokenValue| is  \verb|\SpecialValue| so should have \verb|xxx| in the path, and a file name with two dashes:

\edef\FileWithPath{\GetFileNameWithPath{NonSpecial}{02}{z}}%
\TestFile[blue]{\FileWithPath}

\section{Top directory is special case: Works}
The first parameter is \verb|\SpecialValue|, but not \verb|\the\CurrentTokenValue| so should \textbf{not} have \verb|xxx| in the path, and a file name with two dashes:

\edef\FileWithPath{\GetFileNameWithPath{\SpecialValue}{02}{z}}%
\TestFile[blue]{\FileWithPath}

\section{Both Top directory and TokenValue are special cases}
Since both the first parameter is \verb|\SpecialValue| and \verb|\the\CurrentTokenValue| are a special case, we should \textbf{not} have \verb|xxx| in the path, and a file name should be short:

\CurrentTokenValue={\SpecialValue}
\edef\FileWithPath{\GetFileNameWithPath{\the\CurrentTokenValue}{02}{z}}%
\TestFile{\FileWithPath}

\bigskip
\noindent
Correct result for the above case is
\def\FileWithPath{../special/02/z/z}%

\TestFile[blue]{\FileWithPath}
\end{document}

-
Use a tl (macro) instead of a toks. By Knuth's design, within an \edef, \the\toks... only unpacks the contents of the toks without further expansion. Other option: \expandafter\empty\the\toks.... –  Bruno Le Floch Mar 1 '12 at 6:05
Is that really a expl3 question? –  Marco Daniel Mar 1 '12 at 12:44
@BrunoLeFloch: Seems like that should be an answer and keeps the changes local to GetTexFileName which I like. –  Peter Grill Mar 1 '12 at 16:43
@MarcoDaniel: Don't know. In the area of expansion and expl3 I am a newbie. Feel free to edit the tags. –  Peter Grill Mar 1 '12 at 16:44
@Peter Cf my updated answer to your previs question. I think the whole approach should be changed. It is of course possible to do pretty much anything you want to do expandably (suitable for \edef\path{\getpath{...}{...}{...}}), but you will have much less hoops to go through if you go for a syntax such as \findthepathandsaveit\path{...}{...}{...}. –  Bruno Le Floch Mar 2 '12 at 8:15

I'd go with a token list, but the simple fix

\protected\def\SpecialValue{special}


seems to give the required output: in section 3 I get

Token Value = special

Correct value for this case is
Token Value = special

In the "special" case the comparison is precisely between \SpecialValue and \SpecialValue (not expanded).

A different approach is to avoid token registers:

\ExplSyntaxOn
\tl_new:N \SpecialValue
\tl_set:Nn \SpecialValue {special}
\tl_new:N \l_grill_current_value_tl

\NewDocumentCommand{\CurrentTokenValue}{m}
{
\tl_set:Nn \l_grill_current_value_tl { #1 }
}
\DeclareExpandableDocumentCommand{\GetCurrentTokenValue}{ }
{
\tl_use:N \l_grill_current_value_tl
}
% Determine path: Always #1/xxx#2/#3 unless #1=\SpecialValue
\NewDocumentCommand{\GetTexFilePath}{m m m}
{
\str_if_eq:xxTF{#1}{\SpecialValue}{../#1/#2/#3}{../#1/xxx#2/#3}
}

% Determine file name: Always #1-#2-#3 unless \the\CurrentTokenValue=\SpecialValue.
\NewDocumentCommand{\GetTexFileName}{m m m}
{
\str_if_eq:xxTF{\l_grill_current_value_tl}{\SpecialValue}{#3}{#1-#2-#3}%
}
\ExplSyntaxOff


It's sufficient now to change the syntax \CurrentTokenValue={...} into \CurrentTokenValue{...} and \the\CurrentTokenValue into \GetCurrentTokenValue.

-
Thanks, this works, although I was hoping for a fix localized to GetTexFileName. But, can you add a short explanation of what is going on here, including some information on the downside of using \protected\def. Just wondering what other impact this change may have related to \SpecialValue. –  Peter Grill Mar 1 '12 at 16:40
If you really need to expand into \GetTeXFileName then don't use a token list in the first place: however, you need only to check against the presence of \SpecialValue, not its meaning. –  egreg Mar 1 '12 at 16:50
It would help if, instead of those \edef\FileWithPath you used the main macros along with the expected output. –  egreg Mar 1 '12 at 18:09
Sorry, but in this regards I don't understand what you mean -- I am totally lost when it comes to expansion type of issues. Even after re-reading the posts here on expansion (surprisingly most them actually make sense when I read them). But, when I attempt to solve my problems related to expansion I don't seem to make much progress. It would be helpful if you could add an example in the answer on how not to use the \edef. –  Peter Grill Mar 1 '12 at 18:31