LaTeX macro expander

Question

Is it possible to have LaTeX (or another program) to expand a the macros in a .tex file, and output the expanded LaTeX source?

If no such program exists: are LaTeX macros processed simply as string substitutions, before the document is interpreted? If so, it might be easy enough to write my own such converter.

Edit: The reason I ask is journals which don't accept LaTeX submissions with macros. It would be useful to be able to develop with my usual macros, and convert automatically to the 'macro-expanded' format for submission.

Answer 1

If you just want to see how LaTeX works, and what it does when macros are expanded, you can put \tracingmacros or \tracingall in your .tex file, and LaTeX will write the expansion process in the log file.

Answer 2

Solution for me was http://ctan.org/pkg/de-macro, it is included with TeX Live, then if you are using a Debian based linux distribution it is likely you already have it

what you have to do is:

• place your definitions in a file called something like something-private.sty, in the same directory your original file yourfile.tex is located
• comment or remove your definitions in yourfile.tex
• place a command \usepackage{something-private} in yourfile.tex
• open a shell, and go to the directory where yourfile.tex is located
• in the shell execute de-macro yourfile.tex

a yourfile-clean.tex will be created with the desired macros expanded, if you need to change something in your definitions then remove the *-clean.tex files and run de-macro again.

Answer 3

I'm not aware of a package to do this, but I'd love to have one. The cleveref package has an option along these lines:

When the poorman option is supplied, your document will be processed as normal. But in addition, a sed script will automatically be written, containing rules for replacing all the \cref commands with the LaTeX code that they would produce, and using the standard \ref command to produce the cross-references themselves.

In the package I imagine for LaTeX, you'd have a command such as \newmacro that behaved along the same lines, which would behave like \newcommand except the author would ensure that only "sane" expansions would occur. It could then write an auxiliary file that uses a Lua script to generate a copy of the document with all the macros replaced by their definitions.

So, er, anyone willing to write it?

Answer 4

Using some code meant for upper-casing and lowercasing expandably (which you can grab there, and maybe at some point will become a package on CTAN), it turns out to be possible to produce a file with some macros expanded, with a reasonable speed, and relatively few lines of code.

First, let's see how it is used. Define your macros at the very top of the document (before \documentclass), and set up everything related to the macro expansion. Then the command \ExpandMacros{macroexpander-example-out.tex} grabs the end of the file inside an \edef. The macro thus constructed is then written to the output file. That file is then input back, so that compiling your document still typesets it.

\input macroexpander.tex\relax

% Declare the commands that you want to see expanded between
% |\ExpandingCommands| and |\endExpandingCommands|. Note that
% for the moment, only |\newcommand| is supported, not
% |\newcommand*|, and that optional arguments are not 
% supported (or rather, you'll be surprised by the result).

\ExpandingCommands
\newcommand{\myvec}[3]{(#1,#2,#3)}
\newcommand{\Vcal}{\ensuremath{\mathcal{V}}}
\newcommand{\myitem}{\item[\fbox{$\bullet$}]}
\endExpandingCommands

% You can also force other commands to expand.
\MacexpExpanding\emph % ok, don't do it for |\emph|: I have no good example.

% In order to get a neat output, you can tell Mr. MacXP to put a newline 
% before outputting some commands.
\MacexpNewliner\begin
\MacexpNewliner\end
\MacexpNewliner\title
\MacexpNewliner\author
\MacexpNewliner\maketitle
\MacexpNewliner\item
% This is compatible with the |\newcommand| used above:
\MacexpNewliner\Vcal
% (Note that |\myitem| defined above expands to |\item|, which produces 
% a newline, so we don't want |\MacexpNewliner\myitem|, which would add
% one more line!)
%
% The next line does the work. The argument is the name of the file where we
% shall write.
\ExpandMacros{macroexpander-example-out.tex}%
% ===============
\documentclass{article}
\begin{document}
\title{Example}
\author{Someone}
\maketitle

Let $x {= \myvec{x_1}{x_2}{x_3}}$ {b}e a \emph{nice} element of \Vcal.
Let $x = \myvec{x_1}{x_2}{x_3}$ be a nice element of \Vcal.
\begin{itemize}
\item
\myitem Let $x = \myvec{x_1}{x_2}{x_3}$ be a nice element of \Vcal.
\item Let $x = \myvec{x_1}{x_2}{x_3}$ be a nice element of \Vcal.
\myitem Let {$x = \myvec{x_1}{x_2}{x_3}$ be a nice element of} \Vcal.
\item 
\myitem
\end{itemize}

\end{document}

The code itself is quite long. Basically, ULcase.sty does all the heavy lifting of picking up tokens one by one (particularly spaces and braces). So we just need to decide what we do with each token, by specifying a table of case changes.

A short aside: in my view, there are three types of expandability: (the names are badly chosen... better names, anyone?)

• the "strong" type is when hitting <token-list> with a given number of expansions (with e.g., \expandafter) will expand it fully.

• the "medium" type if \romannumeral-`\0<token-list> expands completely, or equivalently, if after enough expansions (coming from the left) we obtain the full expansion of <token-list>.

• the "weak" type is when \edef\foo{<token-list>} works.

For example, \relax \empty is only weakly expandable, \empty\relax is "strongly" expandable, etc. (note: as Hendrik pointed out, calling \relax expandable is just wrong, so the names should be changed). In our case, we will do \edef\foo{... followed by the text we want to scan. Thus, we only need weak expandability overall. This means that at the outer level of grouping, we can simply output tokens to the left and continue. This will be key for efficiency. However, to treat brace groups, we need to get the full expansion of the content before outputting it. So we use a different "case-change" table, which outputs tokens at the end of the group, so that the expansion coming from the left can continue to act.

\begingroup
    % Code based on the extended Upper- and Lower-casing code found
    % in the ULcase package.
    \input ULcase.sty\relax


    % We use two tables of case change. 
    % 
    % - at the outer level of grouping, we use a table that outputs tokens
    % on the left, |\unexpanded|. This is quick, but it will only work
    % in the context of an |\edef|, |\write|, etc. Otherwise these tokens
    % would go in the way of a further expansion. 
    % 
    % - inside brace groups, we need to expand "from the left" of the group:
    % since we use |\romannumeral| to fully expand, only the left-most macros
    % get expanded. So we need to output tokens by storing them at the end
    % of the group, as we do usually (See |ULcase.sty|).
    % 
    \long\gdef\UL_table_macexp_outer_default{\unexpanded}%
    \long\gdef\UL_table_macexp_outer_braces#1#2{%
      \unexpanded\expandafter\expandafter\expandafter{%
        \expandafter\expandafter\expandafter{\UL_to_case:nn{macexp_inner}{#2}}%
      }% Note that we use the |macexp_inner| table here.
    }
    % For the inner table, we just use the standard output, and keep the
    % same table, |#1|.
    \long\gdef\UL_table_macexp_inner_default{\UL_to_case_output:n}%
    \long\gdef\UL_table_macexp_inner_braces#1#2{%
      \expandafter\expandafter\expandafter\UL_to_case_output:n%
      \expandafter\expandafter\expandafter{%
        \expandafter\expandafter\expandafter{\UL_to_case:nn{#1}{#2}}%
      }%
    }


    %======= Customization.
    % This long macro just initializes the table element |#1| of
    % the tables |macexp_inner| and |macexp_outer| to consist
    % in a |pre-| piece and a |the-| piece, which we configure separately
    % later on (initially |{}|).
    \long\gdef\macexp_set_init:N#1{%
      \ifcsname UL_table_macexp_inner_\detokenize{#1}\endcsname
      \expandafter\use_none:n
      \else
      \expandafter\use:n
      \fi
      {%
        \long\expandafter\gdef\csname UL_table_macexp_inner_%
                                 \detokenize{#1}\endcsname{%
          \csname macexp_inner_pre_\detokenize{#1}\endcsname%
          \csname macexp_inner_the_\detokenize{#1}\endcsname%
        }%
        \long\expandafter\gdef\csname UL_table_macexp_outer_%
                                 \detokenize{#1}\endcsname{%
          \csname macexp_outer_pre_\detokenize{#1}\endcsname%
          \csname macexp_outer_the_\detokenize{#1}\endcsname%
        }%
        \long\expandafter\gdef\csname macexp_inner_pre_%
        \detokenize{#1}\endcsname{}%
        \long\expandafter\gdef\csname macexp_inner_the_%
        \detokenize{#1}\endcsname{\UL_to_case_output:n{#1}}%
        \long\expandafter\gdef\csname macexp_outer_pre_%
        \detokenize{#1}\endcsname{}%
        \long\expandafter\gdef\csname macexp_outer_the_%
        \detokenize{#1}\endcsname{\unexpanded{#1}}%
      }%
    }

    % Declaring a token to be expanding: this defines the |the-| piece 
    % mentionned above for both the inner and outer tables.
    \long\gdef\macexp_set_expanding:N#1{%
      \macexp_set_init:N#1%
      \long\expandafter\gdef\csname macexp_inner_the_%
        \detokenize{#1}\endcsname##1##2{\expandafter:nw{##1{##2}}#1}%
      \long\expandafter\gdef\csname macexp_outer_the_%
        \detokenize{#1}\endcsname##1##2{\expandafter:nw{##1{##2}}#1}%
    }

    % Declaring a token to induce a new-line in the output file
    % (for readability). This is done with the |pre-| piece.
    % 
    \begingroup\catcode`\^^M=12\relax%
    \long\gdef\macexp_set_newliner:N#1{%
      \macexp_set_init:N#1%
      \long\expandafter\gdef\csname macexp_outer_pre_%
        \detokenize{#1}\endcsname{\unexpanded{^^M}}%
      \long\expandafter\gdef\csname macexp_inner_pre_%
        \detokenize{#1}\endcsname{\UL_to_case_output:n{^^M}}%
    }%
    % And we define |\par| to give two lines when it is outside braces.
    \long\expandafter\gdef\csname UL_table_macexp_outer_%
        \detokenize{\par}\endcsname{^^M^^M}%
    \endgroup

    \long\gdef\macexp_newcommand#1{%
      \macexp_set_expanding:N#1%
      \macexp_old_newcommand{#1}%
    }%

    \long\gdef\ExpandingCommands{%
      \let\macexp_old_newcommand\newcommand%
      \let\newcommand\macexp_newcommand}%
    \long\gdef\endExpandingCommands{%
      \let\newcommand\macexp_old_newcommand}%

    \long\gdef\ExpandMacros#1{%
      \begingroup
      \everyeof{%
        {\q_mark} \q_stop\iffalse{\fi}%
        \newlinechar=`\^^M\relax
        \newwrite\macexp_write%
        \immediate\openout\macexp_write=#1\relax%
        \immediate\write\macexp_write{\unexpanded\expandafter{\macro_buffer}}%
        \immediate\closeout\macexp_write%
        \endgroup
        \input #1\relax%
      }%
      \edef\macro_buffer{\iffalse}\fi%
        \UL_brace_check:nw{macexp_outer}%
    }

    % User commands.
    \long\gdef\MacexpNewliner{\macexp_set_newliner:N}
    \long\gdef\MacexpExpanding{\macexp_set_expanding:N}

\endgroup

Answer 5

There's no way to do this in general without also executing the primitive commands since the macros can easily depend on them.

For example, consider

\def\foo{\ifdim\ht0>10pt foo\else bar\fi}
\setbox0\vbox{...}
\foo

Even assuming that the ... was completely specified, \foo could not be expanded until box 0 was set.

It might be possible to completely emulate TeX except for the typesetting and output the stream of tokens, but I'm unaware of such a program.

Answer 6

I've written a perl latex expander that I use for converting standalone latex to wordpress or physicsforums post format. As is it isn't general (i.e. it expands the various macros I use), and doesn't have a pure latex to latex mode (although that wouldn't be too hard).

The workhorse of this script are the following pair of regular expressions for replacing one or two argument commands:

my $r1 = qr/
     (                   # start of capture buffer 1
     {                   # match an opening brace
         (?:
             [^{}]++     # one or more non braces, non backtracking
               |
             (?1)        # found { or }, so recurse to capture buffer 1
         )*
     }                   # match a closing brace
     )                   # end of capture buffer 1
     /x;

my $r2 = qr/
     (                   # start of capture buffer 1
     {                   # match an opening brace
         (?:
             [^{}]++     # one or more non braces, non backtracking
               |
             (?1)        # found { or }, so recurse to capture buffer 1
         )*
     }                   # match a closing brace
     )                   # end of capture buffer 1
     (                   # start of capture buffer 1
     {                   # match an opening brace
         (?:
             [^{}]++     # one or more non braces, non backtracking
               |
             (?1)        # found { or }, so recurse to capture buffer 1
         )*
     }                   # match a closing brace
     )                   # end of capture buffer 1
     /x;

This script has a few issues, and I was toying with the idea of either rewriting it properly with a parser generator or hacking it further. Hoping that somebody else had done something better, I blundered on this question. Without seeing that something better mentioned in any of the other answers, I'll share this in case it has any use to anybody else.

Luca at Stanford also has a latex to wordpress script that could probably be adapted to do latex to latex conversion.

Answer 7

I have developed the latex macro expander in c#. It works very well.

The source file and the excutable program are available on the link:

https://drive.google.com/drive/folders/0B4XhN8fShFOoU0Z3UUxjMGVheXM?usp=sharing

Keywords: Latex newcommand expander; latex macro expander.

For example:

******************************************************"

                Latex newcommands expansion            
                             Version 1.0        
Language: C#
  Usages: 
    1. Drag the latex file on the exe.
    2. LatexNewcommandExpansion.exe file1 file2 ...
       where file1 can be latex file or directory
       contains latex files
 Enjoy!
******************************************************

Example 1:

%test.tex
\newcommand{\v5}[5]{#1 ,#5, \frac{#3}{#4},#2}
\newcommand{\bx}{\mathbf x}

$\bx$ is mathbf x;
$\v5{x}{y}{z}{m}{n}$;

would be

%test_new.tex
\newcommand{\v5}[5]{#1 ,#5, \frac{#3}{#4},#2}
\newcommand{\bx}{\mathbf x}

$\mathbf x$ is mathbf x;
$x ,n, \frac{z}{m},y$;

Example 2:

%test.tex
\newcommand{\v5}[5]{#1 ,#5, \frac{#3}{#4},#2}
\newcommand{\v6}[5]{\bxii{#1}{#5} \v5{#1}{#1}{#2}{#2}{\bx2}}
\newcommand{\bx}{\mathbf x}
\newcommand{\bx2}{\mathbf \bx}
\newcommand{\bxi}{\mathbf y}
\newcommand{\bxi2}[1]{\mathbf{#1}^2}
\newcommand{\bxi3}[1]{\bx\bxi2{#1}}
\newcommand{\bxi4}[1]{\bx2 \bxi3{#1}}
\newcommand{\bxii}[2]{\bxi2{#2}*\exp{\bxi4{#1}}}

%\bx is mathbf x; \bxi is \xi^2  \bxii{x}{y}  \bxi2{z}
\bx is mathbf x; \bxi is \xi^2  \bxii{x}{y}  
\bxi2{z}
\bx2$
\bx2
\bx2\
\bxi2{y}
\bxi4{z}
\bxi3{m}
\v5{x}{y}{z}{m}{n};
\v6{x}{y}{z}{m}{n};

%would be: test_New.tex
\newcommand{\v5}[5]{#1 ,#5, \frac{#3}{#4},#2}
\newcommand{\v6}[5]{\bxii{#1}{#5} \v5{#1}{#1}{#2}{#2}{\bx2}}
\newcommand{\bx}{\mathbf x}
\newcommand{\bx2}{\mathbf \bx}
\newcommand{\bxi}{\mathbf y}
\newcommand{\bxi2}[1]{\mathbf{#1}^2}
\newcommand{\bxi3}[1]{\bx\bxi2{#1}}
\newcommand{\bxi4}[1]{\bx2 \bxi3{#1}}
\newcommand{\bxii}[2]{\bxi2{#2}*\exp{\bxi4{#1}}}

%\bx is mathbf x; \bxi is \xi^2  \bxii{x}{y}  \bxi2{z}
\mathbf x is mathbf x; \mathbf y is \xi^2  \mathbf{y}^2*\exp{\mathbf \mathbf x \mathbf x\mathbf{x}^2}  
\mathbf{z}^2
\mathbf \mathbf x$
\mathbf \mathbf x
\mathbf \mathbf x\
\mathbf{y}^2
\mathbf \mathbf x \mathbf x\mathbf{z}^2
\mathbf x\mathbf{m}^2
x ,n, \frac{z}{m},y;
\mathbf{n}^2*\exp{\mathbf \mathbf x \mathbf x\mathbf{x}^2} x ,\mathbf \mathbf x, \frac{y}{y},x;