Find all broken urls and references in a LaTex document with footnote urls and bibtex urls

Question

Is there some automated way to test broken urls over a LaTex document for errors such as 404 etc. I noticed that some broken urls but the list of reference is too massive to go through one-by-one. Ideas how to solve this kind quality control problem fast?

I have some ideas to solve this:

1. open up writing-process like in SO, with watchful peers, bugs tend become shallow

2. some at least partly automated processing of the document, perhaps something like apache-logging or something in that degree? (this is mere speculation and I have no idea how this could be done for a LaTex document to find and process broken material -- perhaps breaking up the large document into some blog series for peer-review where anyone can comment but it may become a larger problem of data-management)

3. other idea?

Answer 1

Having the same problem I just used the following scripts. Sure some Perl hacker can make it a one-liner ;-). The first extract all explicit URLs (\url{...} from .tex and url = {...} from .bib. I called it as extractlinks.pl *.tex *.bib | sort | uniq > urls.txt to get a list of URLs in a file:

#!/usr/bin/perl
use File::Slurp qw(read_file);
foreach my $file (@ARGV) {
    foreach my $line (read_file($file)) {
        my @urls = ($file =~ /\.bib$/)
            ? $line =~ m/^\s*url\s*=\s*{([^}]+)}/
            : $line =~ m/\\url{([^}]+)}/g;
        print "$_\n" for @urls;
    }
}

The second script tries to download each URL with wget. On success the URL is printed to STDOUT, on failure it is printed to STDERR. I called the script as ./checklinks.sh < urls.txt > url-ok.txt 2> url-fail.txt:

#!/bin/bash
while read url; do
    wget -O/dev/null -q "$url"
    if [ $? -eq 0 ]; then
        echo $url
    else 
        echo $url 1>&2
    fi
done

Answer 2

If your URLs are consistently marked up, eg as \url{...} then it should be easy to extract a full list of them either just using sed or something or redefining \url to write them out. Given such a list it is easy to check all the url link to available documents, you could just use a command line tool like wget or an online link checker like http://validator.w3.org/checklink