13

Is there any way to test the hyperlinks of a pdf file ? Suppose I have a pdf file containing a large number of hyperlinks, can I test automatically the response I will have when following those links (200 = ok, 404 = not found, etc.).

I know such tools exists for webpages (as softwares, like KLinkStatus, or as webservices, like the checklink of the W3C) but is there any equivalent for pdf? As I guess there is no such thing, is there any nice equivalent for tex files? I guess this can be done with some scripts and regexp magic, but I was wondering if there was any existing solution.

\documentclass{article}
\usepackage{hyperref}

\begin{document}
\href{http://tex.stackexchange.com/test-404/}{This will give a 404}

\href{http://tex.stackexchange.com/}{This will give a 200}

\end{document}
7
  • I googled "testing pdf hyperlinks" and found several intersting sites, like axmag.com/manual/PC/Activate_Links.htm Oct 4, 2014 at 4:35
  • This is useless: this website propose to "convert" your PDF files (into what?) and then to "click on the links to see if they are activated." So it is just a useless conversion to end up clicking on every link, like I could do on a pdf file.
    – Clément
    Oct 4, 2014 at 11:17
  • I guess another option would be to use latex2html to produce an html document that could be easily tested, but I'm sure something more natural exists…
    – Clément
    Oct 6, 2014 at 20:40
  • 4
    Wouldn't Super User be a better place for asking this? It's not really related to TeX and friends.
    – egreg
    Jan 12, 2015 at 0:02
  • 1
    I don't understand how this is off-topic, either. I could easily imagine an answer consisting of (hypothetically) an option to hyperref that produces a .txt file with URLs in them and them a quick script (like in the accepted answer) to check the status codes. For a similar style of question/answer, consider the minted questions about using custom lexers and highlighting schemes (which tend to show Python code). Nov 21 at 17:23

3 Answers 3

8

Install curl (assuming you use apt-get to install packages) :

sudo apt-get install curl

and do the following for pdf files:

cat document.pdf | grep -ao '[a-zA-Z]\+://[a-zA-Z.-]*[^)]\+' | while read LINE; do curl -o /dev/null --silent --head --write-out '%{http_code}' "$LINE";    echo " $LINE";  done

and do the following for tex files :

cat myfile.tex | grep -o '{[a-zA-Z]\+://[a-zA-Z.-]*[^}]\+}' | sed s/{// | sed s/}// | while read LINE; do curl -o /dev/null --silent --head --write-out '%{http_code}' "$LINE" echo " $LINE" done

Copy paste the following to your command line for an immediate test:

echo "\documentclass{article}
\usepackage{hyperref}

\begin{document}
\href{http://tex.stackexchange.com/test-404/}{This will give a 404}

\href{http://tex.stackexchange.com/}{This will give a 200}

\end{document}" | grep -o '{[a-zA-Z]\+://[a-zA-Z.-]*[^}]\+}' | sed s/{// | sed s/}// | while read LINE; do
   curl -o /dev/null --silent --head --write-out '%{http_code}' "$LINE"
   echo " $LINE"
 done

Refer to the following for details:

https://stackoverflow.com/questions/13611973/how-to-grep-for-a-url-in-a-file https://stackoverflow.com/questions/6136022/script-to-get-the-http-status-code-of-a-list-of-urls

6
  • @Joseph Wright ... I think it's unfair that the bounty got refunded ;-)
    – Koni
    Jan 13, 2015 at 19:50
  • @Clément at least a tip would be nice ;-)
    – Koni
    Jan 13, 2015 at 19:51
  • 1
    There is no way I can do that. And I am sorry that the bounty has been refused, but cannot do anything to change that.
    – Clément
    Jan 14, 2015 at 12:37
  • Just upvote my solution ;-)
    – Koni
    Jan 14, 2015 at 15:18
  • I already did, of course!
    – Clément
    Jan 14, 2015 at 15:20
3

If you are open to using Python this may suffice.

I use python with the pyPdf and urllib2 packages. The logic goes like this: Open a pdf with pyPdf.PdfFileReader and loop through it. In the function below, pdf is the open pdf file object.

def get_urls(pdf):
    badurls = list()
    links = list()
    url_checker = URLChecker() # a helper function to test the URL (urllib2)
    for pg in range(pdf.getNumPages()):
        page = pdf.getPage(pg)
        obj = page.getObject()

        for a in obj.get('/Annots', []):
            u = a.getObject()
            lnk = u['/A'].get('/D')
            url = u['/A'].get('/URI')
            if lnk:
                links.append(lnk)
            if url:
                urls.append(url)
                result, reason = url_checker.check(url)
                if not result:
                    badurls.append({'url':url, 'reason': '%r' % reason})

    anchors = pdf.getNamedDestinations().keys()
    badlinks = [x for x in links if x not in anchors]
    return urls, badurls, badlinks

The url_checker opens the url and if there is an error, it returns (False, error_string). Otherwise it returns (True, None)

So at the end of the process, you have a a list of urls in the pdf, any bad urls that could not be opened (containing the reason), a list of links in the PDF, and a the subset of those links that do not have target anchors.

I've omitted the logic for the url_checker, but it is pretty simple. There are multiple ways to do that part--you could also use the requests package instead of urllib2.

3
  • 1
    "Open a pdf with pyPdf.PdfFileReader and loop through it." I know some of these words. Please could you be more precise ?
    – Clément
    Oct 9, 2014 at 20:28
  • @Clément, yes it is actually in the code block--see the two for loops? The first (outer loop) loops through the pages in the document, the second (inner loop) loops through the annotations on each given page. Anything on a page that responds to the user is an Annotation. If you write a function to check the url, and pass the function above an open PDF file (using pyPdf lib), the code above will work as-is. If you need more help, please be specific on what is giving you problems. I think this answer won't help you much if you don't already know python.
    – Tim A
    Oct 10, 2014 at 21:31
  • 2
    As you guessed, for someone who does already know python, this comment is of little help. Could you start from the beginning, please: "save this as a file, open a terminal, etc." ?
    – Clément
    Oct 13, 2014 at 14:31
2

A simple C++ program using Qt4 and Poppler would do the trick. I quickly sketched it up and tested it on some PDFs. I think one could tweak it a lot to make it more robust against weird links like mailto: -protocol and other ports. And of course Output formatting can be better, if you need to parse it afterwards, but it does the trick

#include <QTcpSocket>
#include <QUrl>
#include <QByteArray>
#include <QList>
#include <poppler-qt4.h>
#include <QDebug>

  using namespace Poppler;

  QString urlGetStatus(const QUrl &url) {
    QString status;
    QTcpSocket socket;
    socket.connectToHost(url.host(), 80);
    if (socket.waitForConnected()) {
        socket.write("HEAD " + url.path().toUtf8() + " HTTP/1.1\r\n"
                     "Host: " + url.host().toUtf8() + "\r\n"
                     "\r\n");
        if (socket.waitForReadyRead()) {
                QByteArray bytes = socket.readAll();
                status = QString(bytes).mid(9, 3);
        }
    }
    return status;
  }

  int main(int argc, char *argv[]) {
    if(argc < 2) {
        return 1;
    }
    QString path = QString(argv[1]);
    Document *doc = Document::load(path);
    if(doc == NULL)
        return 1;
    QList<QUrl> urlList;
    for(int i = 0;i < doc->numPages(); i++) {        
        Page *p = doc->page(i);
        foreach(Link *li,p->links()) {
            if(li->linkType() == Link::Browse) {
                LinkBrowse *link = static_cast<LinkBrowse*>(li);
                urlList.append(QUrl(link->url()));
            }
        }
    }
    foreach(QUrl url, urlList) {
        QString stat = urlGetStatus(url);
        if(stat == "200") {
            qDebug() << url.toString() << "returned status 200";
        } else {
            qDebug() << url.toString() << " maybe not reachable status" << stat;
        }
    }
    return 0;
  }

As I am a cmake friend I used this CMakeLists.txt to compile:

cmake_minimum_required(VERSION 2.6)
project(qlinkextract)
set (CMAKE_MODULE_PATH "${PROJECT_SOURCE_DIR}/cmake/Modules/")

find_package(Qt4 REQUIRED QtCore QtNetwork)
include(${QT_USE_FILE})

find_package(Poppler REQUIRED)
include_directories(${POPPLER_QT_INCLUDE_DIR})

add_executable(qlinkextract main.cpp)
target_link_libraries(qlinkextract ${QT_LIBRARIES} ${POPPLER_QT_LIBRARIES})

You will need Qt4 development packages and Poppler-Qt4 development packages. If you don't have a FindPoppler.cmake in your CMakeModules directories, go and grab one online.

To compile it make sure in the project directory qlinkextract is (needs to be exactly spelled like the followng)

  • CMakeLists.txt (see above)
  • main.cpp (see above)
  • cmake/Modules/FindPoppler.cmake (download from somewhere)
  • build/ (folder is optional)

on a console go to the build folder and type

cmake ..
make

if something is missing install the missing packages

Some example output:

"http://www.igi-global.com/chapter/ontology-based-multimedia-indexing/42895?camid=4v1" returned status 200 
"http://www.igi-global.com/chapter/ontology-based-multimedia-indexing/42895?camid=4v1" returned status 200 
"http://www.igi-global.com/e-resources/library-recommendation/?id=1" returned status 200 
"http://www.igi-global.com/chapter/towards-low-cost-energy-monitoring/112719?camid=4v1a" returned status 200 
"http://www.igi-global.com/article/algebraic-properties-of-rough-set-on-two-universal-sets-based-on-multigranulation/116046?camid=4v1a" returned status 200 
"http://www.igi-global.com/article/algebraic-properties-of-rough-set-on-two-universal-sets-based-on-multigranulation/116046?camid=4v1a" returned status 200 
"http://www.igi-global.com/article/fuzzy-decision-support-system-for-coronary-artery-disease-diagnosis-based-on-rough-set-theory/111313?camid=4v1a" returned status 200 
"http://www.igi-global.com/article/fuzzy-decision-support-system-for-coronary-artery-disease-diagnosis-based-on-rough-set-theory/111313?camid=4v1a" returned status 200 
"http://www.igi-global.com/chapter/optimization-model-identification-temperature-intelligent/74536?camid=4v1a" returned status 200 
"http://www.igi-global.com/chapter/optimization-model-identification-temperature-intelligent/74536?camid=4v1a" returned status 200 
4
  • Welcome to TeX.SX! Your post provides some answer to a probably off-topic question. It has nothing to do with (La)TeX itself.
    – user31729
    Jan 12, 2015 at 0:04
  • 1
    Yes I agree. But it got asked here, so I thought I could help anyway. WIll it be moved to somewhere else?
    – SeDav
    Jan 12, 2015 at 0:12
  • SeDaV: It might be migrated to Super User, but I did not flag your post as off-topic, since I did not want to get it deleted. The question and all answers should be migrated as a bundle, of course. I think this will happen after the bounty period has ended.
    – user31729
    Jan 12, 2015 at 0:14
  • 1
    I think, it's a helpful post anyway
    – user31729
    Jan 12, 2015 at 0:19

Not the answer you're looking for? Browse other questions tagged or ask your own question.