Is there a script that reads a TeX file and replaces every instance of a \newcommand?

Question

I'd like to know if there's a script that reads a .tex file and replace every instance of a non-standard TeX command with whatever it is replacing. I am not sure if what I want is clear but let me give an example:

Suppose the input is:

\documentclass{amsart} 
\usepackage{amsmath,amssymb}
\newcommand{\N}{\mathbb{N}}
\DeclareMathOperator{\End}{End}

\begin{document}
In this lecture we'll study the ring of Endomorphisms of an Abelian group $A$.
Let's denote this ring by $\End(A)$. Throughout the lecture, $\N$ will denote
the set of natural numbers.
\end{document} 

Then, a desirable output of such a script is:

\documentclass{amsart}
\usepackage{amsmath, amssymb}
\begin{document}
In this lecture we'll study the ring of Endomorphisms of an Abelian group $A$.
Let's denote this ring by $\operatorname{End}(A)$. Throughout the lecture,  
 $\mathbb{N}$ will denote the set of natural numbers.
\end{document}

P.S.: I think I had seen something to this effect but I neither remember the place nor a key word to fire up Google.

---

I meant to write that, all answers here are awesome, but I miscounted 2 for 4. :(

Answer 1

^{I've been forced by the TeX.sx chatroom mafia to post my lovely, buggy, terrible, traumatic, post-apocalyptic poor man's implementation of a replacement script. :)}

Well, sadly this won't be a TeX answer. :) Here's my humble attempt, with a script language I'm terrible at.

_{(I'm looking at you, Python!)}

import re
import sys

if len(sys.argv) != 3:
    print('We need two arguments.')
    sys.exit()

inputHandler = open(sys.argv[1], 'r')

mathDictionary = {}
commandDictionary = {}

print('Extracting commands...')
for line in inputHandler:
    mathOperator = re.search('\\\\DeclareMathOperator{\\\\([A-Za-z]*)}{(.*)}', line)
    if mathOperator:
        mathDictionary[mathOperator.group(1)] = mathOperator.group(2)
    newCommand = re.search('\\\\newcommand{\\\\([A-Za-z]*)}{(.*)}', line)
    if newCommand:
        commandDictionary[newCommand.group(1)] = newCommand.group(2)

inputHandler.seek(0)

print('Replacing occurrences...')
outputHandler = open(sys.argv[2],'w')
for line in inputHandler:
    current = line
    for x in mathDictionary:
        current = re.sub('\\\\DeclareMathOperator{\\\\' + x + '}{(.*)}', '', current)
        current = re.sub('\\\\' + x + '(?!\w)', '\\operatorname{' + mathDictionary[x] + '}', current)
    for x in commandDictionary:
        current = re.sub('\\\\newcommand{\\\\' + x + '}{(.*)}', '', current)
        current = re.sub('\\\\' + x + '(?!\w)', commandDictionary[x], current)
    outputHandler.write(current)

print('Done.')

inputHandler.close()
outputHandler.close()

Now, we simply call it:

$ python myconverter.py input.tex output.tex
Extracting commands...
Replacing occurrences...
Done.

input.tex

\documentclass{amsart} 
\usepackage{amsmath,amssymb}
\newcommand{\N}{\mathbb{N}}
\DeclareMathOperator{\End}{End}

\begin{document}
In this lecture we'll study the ring of Endomorphisms of an Abelian group $A$.
Let's denote this ring by $\End(A)$. Throughout the lecture, $\N$ will denote
the set of natural numbers.
\end{document} 

output.tex

\documentclass{amsart} 
\usepackage{amsmath,amssymb}



\begin{document}
In this lecture we'll study the ring of Endomorphisms of an Abelian group $A$.
Let's denote this ring by $\operatorname{End}(A)$. Throughout the lecture, $\mathbb{N}$ will denote
the set of natural numbers.
\end{document} 

Limitations:

• It's my code, so beware! :)
• It works only with \DeclareMathOperator{...}{...} and \newcommand{...}{...}.
• No optional arguments are supported for \newcommand.
• The declaration must be in only one line.
• Balanced curly brackets, please. :)

I know that regular expressions are not suitable for parsing TeX, but they should work for very simple replacements.

Here's a beautiful reading about regex. Have fun. :)

Answer 2

By accident I came upon de-macro, which is a Python script for such a purpose. It is included in TeX Live.

Limitation: It only affects \newcommand, \renewcommand, \newenvironment and \renewenvironment. Starred versions and optional arguments are not handled. The following is quoted from this answer by Willie Wong to another question, and has more details about the limitations:

Based on the suggestion of Torbjørn T. and cfr, I looked more deeply into the de-macro package. It works to a certain extent. The following are caveats:

• Unlike the what the documentation suggests, the version I have installed creates the database file <filename> instead of <filename>.db. However evidently it tests for <filename>.db as the name for the definitions database. So in its current incarnation it will recreate the definitions database from scratch every single run. For small documents this is not an issue. For larger documents one should copy (not move!) the database to <filename>.db to take advantage of any potential speed-up.

• There are still some bugs to be ironed out. Occasionally it mangles preambles by inserting spurious } to the code. I have not yet found the reason why or the trigger/MWE for it. The small test cases I tried it on all worked fine in this regard.

• Very important: as the documentation suggests, all the definitions you want swapped out must live inside a separate package ending in -private.sty. In the main .tex file one must use that package.

• Also very important: the program handles \newcommand and \renewcommand, but not the starred variant \newcommand* (though this can be fixed by amending the regular expression a bit in the python code, I suppose). This is why my first attempt failed. (I always use the starred variant since I learned about it being best practice.)

• Also very important: after removing the stars, the program threw up an error. Which I eventually figured out is due to my habit of writing \newcommand\cmdname{<replacement} instead \newcommand{\cmdname}{<replacement>}. That extra pair of braces is important for the parsing!

• Lastly, very disappointing for me, the program cannot handle optional arguments. \newcommand{\cmdname}[2]{blah #1 blah #2} works fine, but \newcommand{\cmdname}[2][nothing]{blah #1 blah #2} throws an exception.

The problem with the star and the braces I can easily fix/workaround by rewriting my macro definitions (which as you remember, will be thrown away at the end anyway being the point of this exercise) without the stars and adding the extra braces.

The problem with optional argument handling however, currently makes the program somewhat less useful for me. I can work around it for now by splitting the optional and non-optional commands into two separate ones. Maybe, if I have some time in the future, I will look into adding support for it, after figuring out the logic of the original python script.

Answer 3

Here's a perl script to do the same job. It has the same limitations as Paulo's code, but works well in your test case. I don't doubt it could be improved upon :)

You use it in the following way

perl replacenewcommands.plx myfile.tex

which outputs to the terminal, or

perl replacenewcommands.plx myfile.tex > outputfile.tex

which will output to outputfile.tex

replacenewcommands.plx

#!/usr/bin/perl

use strict;
use warnings;

# for newcommands
my @newcommandmacro=();
my %newcommandcontent=();

# for DeclareMathoperator
my @declaremathoperator=();
my %declaremathoperatorcontent=();

# for use as an index
my $macro;

# loop through the lines in the INPUT file
while(<>)
{
    # check for 
    #   \newcommand...
    # and make sure not to match
    #   %\newcommand
    # which is commented
    if($_ =~ m/\\newcommand{(.*)}{(.*)}/ and $_ !~ m/^%/)
    {
        push(@newcommandmacro,$1);
        $newcommandcontent{$1}=$2;

        # remove the \newcommand from the preamble
        s/\\newcommand.*//;
    }


    # loop through the newcommands in the 
    # main document
    foreach $macro (@newcommandmacro)
    {
      # make the substitution, making sure to escape the \
      # uinsg \Q and \E for begining and end respectively
      s/\Q$macro\E/$newcommandcontent{$macro}/g;
    }

    # check for 
    #   \DeclareMathOperator...
    # and make sure not to match
    #   %\DeclareMathOperator
    # which is commented
    if($_ =~ m/\\DeclareMathOperator{(.*)}{(.*)}/ and $_ !~ m/^%/)
    {
        push(@declaremathoperator,$1);
        $declaremathoperatorcontent{$1}=$2;

        # remove the \DeclareMathOperator from the preamble
        s/\\DeclareMathOperator.*//;
    }

    # loop through the DeclareMathOperators in the 
    # main document
    foreach $macro (@declaremathoperator)
    {
      # make the substitution, making sure to escape the \
      # uinsg \Q and \E for begining and end respectively
      s/\Q$macro\E(\(.*\))/\\operatorname{$declaremathoperatorcontent{$macro}}$1/g;
    }
    print $_;
}

In your test case

myfile.tex (original)

\documentclass{amsart} 
\usepackage{amsmath,amssymb}
\newcommand{\N}{\mathbb{N}}
\newcommand{\mycommand}{something else}
\DeclareMathOperator{\End}{End}

\begin{document}
In this lecture we'll study the ring of Endomorphisms of an Abelian group $A$.
Let's $\N$ denote this ring by $\End(A)$. Throughout the lecture, $\N$ will denote
the set of natural numbers. \mycommand

and \mycommand again
\end{document} 

outputfile.tex (new)

\documentclass{amsart} 
\usepackage{amsmath,amssymb}




\begin{document}
In this lecture we'll study the ring of Endomorphisms of an Abelian group $A$.
Let's $\mathbb{N}$ denote this ring by $\operatorname{End}(A)$. Throughout the lecture, $\mathbb{N}$ will denote
the set of natural numbers. something else

and something else again
\end{document} 

Answer 4

This answer is a complement to Paulo Cereda's answer

As it stands, the regex he uses doesn't support nested brackets, and, indeed, the language doesn't allow for nested, recursive structures of arbitrary depth.

That said, we can robustify the regex for one additional level of nesting, like this:

r"\\newcommand{\\([A-Za-z]*)}{((:?[^{}]*{[^}]*})*[^}]*)}"
**             **              |****************|
#1             #2              #3

Notes:

1. The Python r"..." construction for regexes is helpful because it doesn't need Python's text escaping (though it still requires regex escaping).
2. That means we only need \ to represent \ in the regex, making us all less cross-eyed
3. The main addition is a subpattern that looks for zero or more balanced brackets, followed by the tail end of the original pattern.

It's also possible to improve efficiency slightly by pre-compiling some of the regexes. Here's my adaptation of Paulo's solution:

#! python.exe

import re
import sys

if len(sys.argv) != 3:
    print('We need two arguments.')
    sys.exit()

mathDictionary = {}
commandDictionary = {}

inputHandler = open(sys.argv[1], 'r')
outputHandler = open(sys.argv[2],'w')

mathpat    = re.compile( r"\\DeclareMathOperator{\\([A-Za-z]*)}{((:?[^{}]*{[^}]*})*[^}]*)}" )
searchmath = mathpat.search 
newcpat    = re.compile( r"\\newcommand{\\([A-Za-z]*)}{((:?[^{}]*{[^}]*})*[^}]*)}" )
searchnewc = newcpat.search 

print('Processing...')

for line in inputHandler:
    mathOperator = searchmath(line)
    if mathOperator:
        mathDictionary[mathOperator.group(1)] = mathOperator.group(2)
    newCommand = searchnewc(line)
    if newCommand:
        commandDictionary[newCommand.group(1)] = newCommand.group(2)
    current = line
    for x in mathDictionary:
        current = re.sub(mathpat, '', current)
        current = re.sub(r"\\" + x + r"(?!\w)", '\\operatorname{' + mathDictionary[x] + '}', current)
    for x in commandDictionary:
        current = re.sub(newcpat, '', current)
        current = re.sub(r"\\" + x + "(?!\w)", commandDictionary[x], current)
    outputHandler.write(current)

inputHandler.close()
outputHandler.close()

print('Done.')

I have also made it one-pass; defining a newcommand after using it doesn't make too much sense.

Answer 5

I wrote some javascript for expanding macros defined by \def, \gdef, \edef, \xdef, \newcommand, \newcommand*, \renewcommand, \renewcommand*, \DeclareMathOperator and \DeclareMathOperator*. You may try it here.

function expandMacros(tex) {
    function nestBrackets(level) {
        var level = level || 5, re = c = "(?:[^\\r\\n\\{\\}]|\\\\[\\{\\}]|\\r?\\n(?!\\r?\\n))*?";
        while (level--) re = c + "(?:\\{" + re + "\}" + c + ")*?";
        return " *(\\{" + re + "\\}|[^\\{])";
    }    
    function getRegExp(name, macro) {
        var num = macro.num, def = macro.def, re = "";
        while (num--) re += nestBrackets();
        re = "\\" + name + "(?![a-zA-Z\\}])" + re;
        return new RegExp(re, "g");
    }
    function trimString(s) {
        return s.replace(/^ +| +$/g, '').replace(/^\{|\}$/g, "");
    }
    function extractMacros() {
        var cs = "\\\\\\w+", re;
        // \def, \gdef, \edef and \xdef
        re = new RegExp("\\\\[gex]?def\\*? *(" + cs + ") *(#\\d)*" + nestBrackets(), "g");
        tex = tex.replace(re, function(match){
            var m = arguments;
            var macro = {
                num:  m[2] ? Math.min(m[2].length / 2, 9) : 0,
                def:  trimString(m[3])
            };
            macros[trimString(m[1])] = macro;
            return "";
        });
        // \newcommand, \newcommand*, \renewcommand and \renewcommand*
        re = new RegExp("\\\\(?:re)?newcommand\\*? *(" + cs + "|\\{" + cs + "\}) *(\\[(\\d)\\])?"
                        + nestBrackets(), "g");
        tex = tex.replace(re, function(match){
            var m = arguments;
            var macro = {
                num:  m[3] || 0,
                def:  trimString(m[4])
            };
            macros[trimString(m[1])] = macro;
            return "";
        });
        // \DeclareMathOperator and \DeclareMathOperator* inside amsmath
        re = new RegExp("\\\\DeclareMathOperator(\\*?) *(" + cs + "|\\{" + cs + "\}) *"
                        + nestBrackets(), "g");
        tex = tex.replace(re, function(match){
            var m = arguments;
            var macro = {
                num:  0,
                def:  "\\operatorname" + m[1] + "{" + trimString(m[3]) + "}"
            };
            macros[trimString(m[2])] = macro;
            return "";
        });
    }
    function replaceMacros() {
        var i = 0, m, re, num;
        for (name in macros) {
            m = macros[name];
            re = getRegExp(name, m), num = m.num;
            //console.log(re);
            tex = tex.replace(re, function(match){
                //console.log(arguments);
                var args = [], result = m.def, k;
                for (k = 1; k <= num; k++) {
                    args[k] = trimString(arguments[k]);
                }
                //console.log(args);
                for (k = 1; k <= num; k++) {
                    result = result.replace(new RegExp("#" + k, "g"), args[k]);
                }
                return result;
            });
        }
    }
    var macros = {};
    extractMacros();
    replaceMacros();
    return tex;
}

document.getElementById("run").onclick = function() {
    var input = document.getElementById("input"),
        output = document.getElementById("output");
    output.value = expandMacros(input.value);
}