Utility to Strip Comments from LaTeX Source

I've a LaTeX source.
%But first I would like

So I hope there are
100\% auto
ways to get this done.

\begin{comment}
Because there are subtle ways to mess it up.
\end{comment}


Is there a utility which will eliminate all these comments?

Yes, I could do it by hand, but that seems needlessly laborious, has the potential for mistakes, and makes maintenance difficult. I could also use sed, but there's a potential for mistakes. Besides, it is an axiom of the whole GNU/Linux thing that if you can think of it, someone's probably already made a utility for it.

-
should the text "Because there ..." be printed? –  Herbert Nov 21 '12 at 12:20
No, it's in a comment environment from the verbatim package. –  Richard Nov 21 '12 at 12:29
ah, I see, you want to delete the %blabla .. –  Herbert Nov 21 '12 at 12:32
you might want to try tex.stackexchange.com/questions/82972/… but it would need extending for your examples:-) –  David Carlisle Nov 21 '12 at 12:59
Thanks @DavidCarlisle, alas, I am using the verbatim package, which is not covered by that answer and am not using emacs, so the script doesn't seem to be of direct benefit. –  Richard Nov 21 '12 at 13:01

I'm not sure how to do this. So, I'm posting a new solution. The code I posted yesterday will eat comments from within a verbatim environment.

Here's a new example file to be cleaned:

I've a LaTeX source.
%But first I would like

So I hope there are
100\% auto
ways to get this done.

\begin{comment}
Because there are subtle ways to mess it up.
\end{comment}

\begin{verbatim}
next two lines should not be lost
% don't lose this line
% this line should stay too
\end{verbatim}


According to the verbatim package documentation verbatim and comment environments should not be nested. The following code (similar to what I posted yesterday) will not eat commented lines that appear within a verbatim environment.

Here is the corrected Perl code:

#!/usr/bin/perl
use strict 'vars';
&MAIN(@ARGV);
sub MAIN {
my ($filehandle) = @_; open FILE, "<$filehandle";
my @doc = <FILE>;
close FILE;

foreach my $line ( @doc ){ print$line;
}

return 1;
}

my ($docarray) = @_; my$isCommentEnvironment  = "no";
my $isVerbatimEnvironment = "no"; my @newdoc; foreach my$line ( @{$docarray} ){$isVerbatimEnvironment = "yes" if ( $line =~ /^\\begin{verbatim}/ );$isCommentEnvironment  = "yes" if ( $line =~ /^\\begin{comment}/ ); if ( ($isVerbatimEnvironment eq "no") && ($isCommentEnvironment eq "no") ){ next if ($line =~ /^%/);
## Temporarily replace "%" that you want to keep with a dummy string
## that does not appear elsewhere in your document.  Then, remove remainder
## of lines that still contain "%".
if ( $line =~ /\\%/){$line =~ s/\\%/TMP::PERCENT/g;
$line =~ s/%.*//;$line =~ s/TMP::PERCENT/\\%/g;
} else {
## do not remove trailing % marking NO SPACE in LaTeX: $line =~ s/%.*//;$line =~ s/\s*%.+//;
}
push @newdoc, $line; } push @newdoc,$line if ( $isVerbatimEnvironment eq "yes" );$isVerbatimEnvironment = "no" if ( $line =~ /^\\end{verbatim}/ );$isCommentEnvironment  = "no" if ( $line =~ /^\\end{comment}/ ); } @{$docarray} = @newdoc;
return 1;
}

-
I think the standard convention here on TeX.SX would be to add this as an alternative in the first solution you provide, by using the edit button. –  Andrew Swann Nov 22 '12 at 16:26
How can I output the code shown in the console to a file? –  Jörg Dec 3 '12 at 13:45
Save the script to a file, let's call it stripcomments.pl. And let's say the file you want to remove comments called is mytext.tex. Then on the command line write, perl stripcomments.pl mytext.tex > no_comment.tex. Just be careful. Redirecting output this way will not warn you about overwriting an already existing file. –  A.Ellett Dec 3 '12 at 14:00
The code provided by A.Ellett worked great for me, with a notable exception: the '%' sign at the end of the line should not be removed, as they have the side-effect of "eating" the white space represented by the newline. As a consequence, I used a different regex for the substitution: s/\s*%.+//. –  user1366204 Jul 5 '13 at 22:41
I think also the line next if ($line =~ /^%/); had better be next if ($line =~ /^\s*%/);, else the output will contain blank lines that aren't present in the original –  D. Savitt Nov 19 '14 at 4:18

It can be done using the Python ply.lex module to write a simple tokenizer:

import ply.lex, argparse, io

#Usage
# python stripcomments.py input.tex > output.tex
# python stripcomments.py input.tex -e encoding > output.tex

tokens = (
'PERCENT', 'BEGINCOMMENT', 'ENDCOMMENT', 'BACKSLASH',
'CHAR', 'BEGINVERBATIM', 'ENDVERBATIM', 'NEWLINE', 'ESCPCT',
)
states = (
('linecomment', 'exclusive'),
('commentenv', 'exclusive'),
('verbatim', 'exclusive')
)

#Deal with escaped backslashes, so we don't think they're escaping %.
def t_ANY_BACKSLASH(t):
r"\\\\"
return t

def t_PERCENT(t):
r"\%"
t.lexer.begin("linecomment")

#Escaped percent signs
def t_ESCPCT(t):
r"\\\%"
return t

#Comment environment, as defined by verbatim package
def t_BEGINCOMMENT(t):
r"\\begin\s*{\s*comment\s*}"
t.lexer.begin("commentenv")

#Verbatim environment (different treatment of comments within)
def t_BEGINVERBATIM(t):
r"\\begin\s*{\s*verbatim\s*}"
t.lexer.begin("verbatim")
return t

#Any other character in initial state we leave alone
def t_CHAR(t):
r"."
return t

def t_NEWLINE(t):
r"\n"
return t

#End comment environment
def t_commentenv_ENDCOMMENT(t):
r"\\end\s*{\s*comment\s*}"
#Anything after \end{comment} on a line is ignored!
t.lexer.begin('linecomment')

def t_commentenv_CHAR(t):
r"."
pass

def t_commentenv_NEWLINE(t):
r"\n"
pass

#End of verbatim environment
def t_verbatim_ENDVERBATIM(t):
r"\\end\s*{\s*verbatim\s*}"
t.lexer.begin('INITIAL')
return t

#Leave contents of verbatim environment alone
def t_verbatim_CHAR(t):
r"."
return t

def t_verbatim_NEWLINE(t):
r"\n"
return t

#End a % comment when we get to a new line
def t_linecomment_ENDCOMMENT(t):
r"\n"
t.lexer.begin("INITIAL")
#Newline at the end of a line comment is stripped.

#Ignore anything after a % on a line
def t_linecomment_CHAR(t):
r"."
pass

lexer = ply.lex.lex()
lexer.input(source)
return u"".join([tok.value for tok in lexer])

def main():
parser = argparse.ArgumentParser()

args = parser.parse_args()

with io.open(args.filename, encoding=args.encoding) as f:

if __name__ == '__main__':
main()


Alternately, here is a Gist.

This correctly handles comments which directly follow a double backslash, such as

Line text \\%comment-text


The first answer above seems to handle this incorrectly (as if the percent sign were escaped), though I don't have sufficient reputation to comment.

-
Welcome to TeX.SX! Rather than just linking to a gist, can you also include the relevant code in your answer? On TeX.SX, we try to avoid link-only answers if we can, since it's possible that the link could break or go dead at some point in the future. –  Adam Liter Nov 30 '14 at 4:32
Thanks for the tip. I've added my code. –  Adam Merberg Nov 30 '14 at 5:32

The following Perl script should do the trick: it expects to receive the name of the file you want stripped of comments and prints to console the modified document.

#!/usr/bin/perl
use strict 'vars';
&MAIN(@ARGV);
sub MAIN {
my ($filehandle) = @_; open FILE, "<$filehandle";
my @doc = <FILE>;
close FILE;

foreach my $line ( @doc ){ print$line;
}
return 1;
}
my ($docarray) = @_; my$isCommentEnvironment = "no";
my @newdoc;

foreach my $line ( @{$docarray} ){
$isCommentEnvironment = "yes" if ($line =~ /^\\begin{comment}/ );
if ( $isCommentEnvironment eq "no" ){ next if ($line =~ /^%/);
## Temporarily replace "%" that you want to keep with a dummy string
## that does not appear elsewhere in your document.  Then, remove remainder
## of lines that still contain "%".
if ( $line =~ /\\%/){$line =~ s/\\%/TMP::PERCENT/g;
$line =~ s/%.*//;$line =~ s/TMP::PERCENT/\\%/g;
} else {
$line =~ s/%.*//; } push @newdoc,$line;
}
$isCommentEnvironment = "no" if ($line =~ /^\\end{comment}/ );
}

@{\$docarray} = @newdoc;
return 1;
}


The only thing I'm uncertain about is any contraints on using \begin{comment} and \end{comment}. For example, I don't know whether---like with the verbatim environment---you are restricted on what else may appear on those lines.

-

pandoc can do this. Running:

pandoc latex-comments.tex -o latex-stripped.tex


Converts latex-comments.tex:

I've a LaTeX source.
%But first I would like

So I hope there are
100\% auto
ways to get this done.

\begin{comment}
Because there are subtle ways to mess it up.
\end{comment}

\begin{verbatim}
next two lines should not be lost
% don't lose this line
% this line should stay too
\end{verbatim}


to latex-stripped.tex:

I've a LaTeX source. I'm ready to strip its comments.

So I hope there are 100\% auto ways to get this done.

Because there are subtle ways to mess it up.

\begin{verbatim}
next two lines should not be lost
% don't lose this line
% this line should stay too
\end{verbatim}


It does remove the comment environment, while leaving the contents unaltered. I'm not sure how these environments are used, but it may be possible to keep them with appropriate config settings in pandoc.

-
Unfortunately, > pandoc latex-comments.tex -o latex-stripped.tex does more than just strip the comments. It also removes most Latex commands and the preamble. –  user60642 Aug 13 '14 at 7:44
@KurtKenner with the -s option it will recreate a stand-alone tex document, with preamble. But I'm not sure how well it incorporates the existing preamble into this :( –  Tyler Aug 13 '14 at 14:44

(Posting the following as an answer rather than a comment to Ellett's Nov 22 '12 at 15:23 answer because the line breaks in code don't show up in comments.) Remarking that latex code of the following form:

here is some text%comment %comment , and here is some more text

is problematic because it compiles as here is some text, and here is some more text whereas the stripped code:

here is some text , and here is some more text

compiles as here is some text , and here is some more text.

(This goes to the same sort of issue about which user1366204 commented.)

-