Is there a script that reads a TeX file and replaces every instance of a \newcommand?

Question

I'd like to know if there's a script that reads a .tex file and replace every instance of a non-standard TeX command with whatever it is replacing. I am not sure if what I want is clear but let me give an example:

Suppose the input is:

\documentclass{amsart} 
\usepackage{amsmath,amssymb}
\newcommand{\N}{\mathbb{N}}
\DeclareMathOperator{\End}{End}

\begin{document}
In this lecture we'll study the ring of Endomorphisms of an Abelian group $A$.
Let's denote this ring by $\End(A)$. Throughout the lecture, $\N$ will denote
the set of natural numbers.
\end{document} 

Then, a desirable output of such a script is:

\documentclass{amsart}
\usepackage{amsmath, amssymb}
\begin{document}
In this lecture we'll study the ring of Endomorphisms of an Abelian group $A$.
Let's denote this ring by $\operatorname{End}(A)$. Throughout the lecture,  
 $\mathbb{N}$ will denote the set of natural numbers.
\end{document}

P.S.: I think I had seen something to this effect but I neither remember the place nor a key word to fire up Google.

---

I meant to write that, all answers here are awesome, but I miscounted 2 for 4. :(

Answer 1

^{I've been forced by the TeX.sx chatroom mafia to post my lovely, buggy, terrible, traumatic, post-apocalyptic poor man's implementation of a replacement script. :)}

Well, sadly this won't be a TeX answer. :) Here's my humble attempt, with a script language I'm terrible at.

_{(I'm looking at you, Python!)}

import re
import sys

if len(sys.argv) != 3:
    print('We need two arguments.')
    sys.exit()

inputHandler = open(sys.argv[1], 'r')

mathDictionary = {}
commandDictionary = {}

print('Extracting commands...')
for line in inputHandler:
    mathOperator = re.search('\\\\DeclareMathOperator{\\\\([A-Za-z]*)}{(.*)}', line)
    if mathOperator:
        mathDictionary[mathOperator.group(1)] = mathOperator.group(2)
    newCommand = re.search('\\\\newcommand{\\\\([A-Za-z]*)}{(.*)}', line)
    if newCommand:
        commandDictionary[newCommand.group(1)] = newCommand.group(2)

inputHandler.seek(0)

print('Replacing occurrences...')
outputHandler = open(sys.argv[2],'w')
for line in inputHandler:
    current = line
    for x in mathDictionary:
        current = re.sub('\\\\DeclareMathOperator{\\\\' + x + '}{(.*)}', '', current)
        current = re.sub('\\\\' + x + '(?!\w)', '\\operatorname{' + mathDictionary[x] + '}', current)
    for x in commandDictionary:
        current = re.sub('\\\\newcommand{\\\\' + x + '}{(.*)}', '', current)
        current = re.sub('\\\\' + x + '(?!\w)', commandDictionary[x], current)
    outputHandler.write(current)

print('Done.')

inputHandler.close()
outputHandler.close()

Now, we simply call it:

$ python myconverter.py input.tex output.tex
Extracting commands...
Replacing occurrences...
Done.

input.tex

\documentclass{amsart} 
\usepackage{amsmath,amssymb}
\newcommand{\N}{\mathbb{N}}
\DeclareMathOperator{\End}{End}

\begin{document}
In this lecture we'll study the ring of Endomorphisms of an Abelian group $A$.
Let's denote this ring by $\End(A)$. Throughout the lecture, $\N$ will denote
the set of natural numbers.
\end{document} 

output.tex

\documentclass{amsart} 
\usepackage{amsmath,amssymb}



\begin{document}
In this lecture we'll study the ring of Endomorphisms of an Abelian group $A$.
Let's denote this ring by $\operatorname{End}(A)$. Throughout the lecture, $\mathbb{N}$ will denote
the set of natural numbers.
\end{document} 

Limitations:

• It's my code, so beware! :)
• It works only with \DeclareMathOperator{...}{...} and \newcommand{...}{...}.
• No optional arguments are supported for \newcommand.
• The declaration must be in only one line.
• Balanced curly brackets, please. :)

I know that regular expressions are not suitable for parsing TeX, but they should work for very simple replacements.

Here's a beautiful reading about regex. Have fun. :)

Answer 2

By accident I came upon de-macro, which is a Python script for such a purpose. It is included in TeX Live.

Limitation: It only affects \newcommand, \renewcommand, \newenvironment and \renewenvironment.

Answer 3

Here's a perl script to do the same job. It has the same limitations as Paulo's code, but works well in your test case. I don't doubt it could be improved upon :)

You use it in the following way

perl replacenewcommands.plx myfile.tex

which outputs to the terminal, or

perl replacenewcommands.plx myfile.tex > outputfile.tex

which will output to outputfile.tex

replacenewcommands.plx

#!/usr/bin/perl

use strict;
use warnings;

# for newcommands
my @newcommandmacro=();
my %newcommandcontent=();

# for DeclareMathoperator
my @declaremathoperator=();
my %declaremathoperatorcontent=();

# for use as an index
my $macro;

# loop through the lines in the INPUT file
while(<>)
{
    # check for 
    #   \newcommand...
    # and make sure not to match
    #   %\newcommand
    # which is commented
    if($_ =~ m/\\newcommand{(.*)}{(.*)}/ and $_ !~ m/^%/)
    {
        push(@newcommandmacro,$1);
        $newcommandcontent{$1}=$2;

        # remove the \newcommand from the preamble
        s/\\newcommand.*//;
    }


    # loop through the newcommands in the 
    # main document
    foreach $macro (@newcommandmacro)
    {
      # make the substitution, making sure to escape the \
      # uinsg \Q and \E for begining and end respectively
      s/\Q$macro\E/$newcommandcontent{$macro}/g;
    }

    # check for 
    #   \DeclareMathOperator...
    # and make sure not to match
    #   %\DeclareMathOperator
    # which is commented
    if($_ =~ m/\\DeclareMathOperator{(.*)}{(.*)}/ and $_ !~ m/^%/)
    {
        push(@declaremathoperator,$1);
        $declaremathoperatorcontent{$1}=$2;

        # remove the \DeclareMathOperator from the preamble
        s/\\DeclareMathOperator.*//;
    }

    # loop through the DeclareMathOperators in the 
    # main document
    foreach $macro (@declaremathoperator)
    {
      # make the substitution, making sure to escape the \
      # uinsg \Q and \E for begining and end respectively
      s/\Q$macro\E(\(.*\))/\\operatorname{$declaremathoperatorcontent{$macro}}$1/g;
    }
    print $_;
}

In your test case

myfile.tex (original)

\documentclass{amsart} 
\usepackage{amsmath,amssymb}
\newcommand{\N}{\mathbb{N}}
\newcommand{\mycommand}{something else}
\DeclareMathOperator{\End}{End}

\begin{document}
In this lecture we'll study the ring of Endomorphisms of an Abelian group $A$.
Let's $\N$ denote this ring by $\End(A)$. Throughout the lecture, $\N$ will denote
the set of natural numbers. \mycommand

and \mycommand again
\end{document} 

outputfile.tex (new)

\documentclass{amsart} 
\usepackage{amsmath,amssymb}




\begin{document}
In this lecture we'll study the ring of Endomorphisms of an Abelian group $A$.
Let's $\mathbb{N}$ denote this ring by $\operatorname{End}(A)$. Throughout the lecture, $\mathbb{N}$ will denote
the set of natural numbers. something else

and something else again
\end{document} 

Answer 4

This answer is a complement to Paulo Cereda's answer

As it stands, the regex he uses doesn't support nested brackets, and, indeed, the language doesn't allow for nested, recursive structures of arbitrary depth.

That said, we can robustify the regex for one additional level of nesting, like this:

r"\\newcommand{\\([A-Za-z]*)}{((:?[^{}]*{[^}]*})*[^}]*)}"
**             **              |****************|
#1             #2              #3

Notes:

1. The Python r"..." construction for regexes is helpful because it doesn't need Python's text escaping (though it still requires regex escaping).
2. That means we only need \ to represent \ in the regex, making us all less cross-eyed
3. The main addition is a subpattern that looks for zero or more balanced brackets, followed by the tail end of the original pattern.

It's also possible to improve efficiency slightly by pre-compiling some of the regexes. Here's my adaptation of Paulo's solution:

#! python.exe

import re
import sys

if len(sys.argv) != 3:
    print('We need two arguments.')
    sys.exit()

mathDictionary = {}
commandDictionary = {}

inputHandler = open(sys.argv[1], 'r')
outputHandler = open(sys.argv[2],'w')

mathpat    = re.compile( r"\\DeclareMathOperator{\\([A-Za-z]*)}{((:?[^{}]*{[^}]*})*[^}]*)}" )
searchmath = mathpat.search 
newcpat    = re.compile( r"\\newcommand{\\([A-Za-z]*)}{((:?[^{}]*{[^}]*})*[^}]*)}" )
searchnewc = newcpat.search 

print('Processing...')

for line in inputHandler:
    mathOperator = searchmath(line)
    if mathOperator:
        mathDictionary[mathOperator.group(1)] = mathOperator.group(2)
    newCommand = searchnewc(line)
    if newCommand:
        commandDictionary[newCommand.group(1)] = newCommand.group(2)
    current = line
    for x in mathDictionary:
        current = re.sub(mathpat, '', current)
        current = re.sub(r"\\" + x + r"(?!\w)", '\\operatorname{' + mathDictionary[x] + '}', current)
    for x in commandDictionary:
        current = re.sub(newcpat, '', current)
        current = re.sub(r"\\" + x + "(?!\w)", commandDictionary[x], current)
    outputHandler.write(current)

inputHandler.close()
outputHandler.close()

print('Done.')

I have also made it one-pass; defining a newcommand after using it doesn't make too much sense.