Grabbing data from website and use it in LaTeX document

This is a follow-up question to "Line breaking and alignment in list".

\documentclass[
a4paper,
12pt
]{article}

\usepackage[
hmargin=2.4cm,
vmargin=3cm
]{geometry}
\usepackage{amssymb}
\usepackage{enumitem}
\usepackage{totcount}
\usepackage[
urlcolor=black
]{hyperref}

\newcommand*\film[4]{%
\item[\refstepcounter{enumi}\textcolor{blue}{$\maltese$} \textsf{[#4]}] \textsf{\href{#1}{#2}~(#3)}}

\regtotcounter{enumi}

\begin{document}

\begin{enumerate}[leftmargin=5em,labelindent=-5em]
\film{http://www.imdb.com/title/tt0134273/}{8MM}{1999}{6,4}
\film{http://www.imdb.com/title/tt0050083/}{12 Angry Men}{1957}{8,9}
\film{http://www.imdb.com/title/tt0057181/}{Incredibly Strange Creatures Who Stopped Living and Became Mixed-Up Zombies, The}{2006}{2,1}
\end{enumerate}

I have \total{enumi}~films.

\end{document}


Now that David has helped me getting the vertical alignment in order, I would like to make the document more automatic: It is possible to grab the film rating from the IMDB site and insert it in the document as the last argument in the \film command? Can I somehow automatically get, e.g., "6.4" from http://www.imdb.com/title/tt0134273/ and insert it in the document?

The point is that the rating changes over time and then I don't have to update the entries manually for each film.

• This is where I would take advantage of *TeX systems' automation possibilities. In your scenario, I would write a simple [Python] script to generate the guts of a LaTeX document, inserting it into a template file I prepare by hand. It would be faster and much more direct. (You can also obviously call latex from such languages if you wish, but a makefile works just as well and maintains modularity.) – Sean Allred Nov 8 '13 at 5:21
• @SeanAllred I'm no 'coding' guy so that is not really a possibility for me. :( – Svend Tveskæg Nov 8 '13 at 5:35
• I'm actually working on a Python solution at the moment; perhaps you'll be able to extend it. I'll comment like crazy, but I would think this is easier than doing it in TeX. – Sean Allred Nov 8 '13 at 5:38
• @SeanAllred That sound very nice but I have a feeling I'm not abel to extend it unless it is very trivial. – Svend Tveskæg Nov 8 '13 at 5:53
• It is very trivial; I'm done the implementation but the writeup remains. – Sean Allred Nov 8 '13 at 6:18

git clone https://github.com/vermiculus/imdb-tex.git && make install && make demo


It's a lot of fun to see how versatile LaTeX is in this respect; I can imagine a version of this script that incorporates the movie poster as well. A complete, compilable example is below; it will create the necessary files. After you typeset the document once (to create these files), run make all to see it fly. I'll explain what I do in the paragraphs that follows.

I would note that, in this case and cases like it, questions like this cross with StackOverflow—all you need to do is insert formatted data into a file.

One of the greatest strengths of TeX systems is its incredible ability to be automatically generated and then processed into a readable document. (This is, I imagine, how train schedules are typeset.) The possibilities are amazing, actually; you could (using IMDB's API, if it exists) easily create a system that would generate a report of every single movie in the database. I don't recommend you do this; the result would be huge ;-).

I didn't mess with any APIs or fancy stuff in this; the following is just a proof-of-concept. The idea stems from the fact that, like LaTeX documents, (X)HTML documents are innately structured. Using Python (or your favorite language), we can take advantage of this structure to search for key pieces of data in the website. We'll create a means to search text for key data (by far the hardest part of the system), and then create and download the source code from URL we generate from a movie identifier. We'll then simply open up our TeX file, find where we wanted to insert our data, insert it using the 'key data' function we made, and close the file. We'll be able to use the script with a simple

./imdb.py <template file> <movie ids...>


Let's create a file called imdb.py and make it executable:

touch imdb.py
chmod +x imdb.py


and tell it how to run itself in the very first line:

#!/usr/bin/env python


(Note this will only work on UNIX systems; make the necessary modifications for Windows if you do not have a UNIX emulation solution like MinGW or Cygwin.)

We define a function that will grab the text inside specific delimiters. (You can imagine a scenario such as <filmrating>6.4</filmrating>, although in this case it's not so expressive. Silly IMDB.)

def get_middle(s, start, end):
import re
return re.search(re.escape(start) + '(.*?)' + re.escape(end), s).group(1)


This function uses regular expressions (re) to search for some <start>...<end> pattern. We group the ... in parentheses and return only that text. You can learn more about regular expressions from a quick Google search, although I recommend adding python to that query; REs are a CS-theoretical topic by nature.

Next, we want to create a function that will create a URL from a movie ID we give it. (Note that the movie ID is the title/... in http://www.imdb.com/title/tt0134273; thus tt0134273.) This is simpler than the previous function still:

def make_url(movie_id):
return 'http://www.imdb.com/title/{mid}'.format(
mid=movie_id)


Since this is an internet-enabled solution, we should create a function that will grab the HTML stored at some URL (perhaps the one we just generated). We'll create a function to do just this:

def get_source(movie_id):
import urllib2
url = make_url(movie_id)
stream = urllib2.urlopen(url)


Simple, right? Now we just want to define a couple 'good reading' functions; functions that only exist to make the code more readable. These functions get different metadata, and I may in a future version accumulate this into a single function with a nicer interface.

def get_rating(source):
return get_middle(source, '<span itemprop="ratingValue">', '</span>')

def get_title(source):
return get_middle(source, '<span class="itemprop" itemprop="name">','</span>')

def get_release(source):
return get_middle(source, '<meta itemprop="datePublished" content="','" />')


Now we just create a function to handle the heavy lifting and orchestrate everything that we've written so far. We create a function that takes a movie ID and a template file and an optional marker.

1. We want to open the file first, reading the file into a list of lines (file_lines).
2. We then find our marker in the file.
3. We download the page for the movie from IMDB and store it in a variable called source.
4. We create a variable, data, to hold the line we want to insert and format it with all of our metadata, using the 'readability functions' we created earlier.
5. We insert data right before the marker (so the marker still exists beneath the line we just inserted).
6. We're done! Write the file.
def insert(movie_id, file_name, marker='%! FILMDATA !%\n'):
"""Insert information for this movie-id before the %! FILMDATA !% line in file_name.

This function should not delete the marker line.
"""
with open(file_name, 'r') as f:
file_lines = list(f)

marker_line = file_lines.index(marker)

source = get_source(movie_id)
data = '\\film{%s}{%s}{%s}{%s}\n' % (
make_url(movie_id),
get_title(source),
get_release(source),
get_rating(source))

file_lines.insert(marker_line, data)

with open(file_name, 'w') as f:
f.writelines(file_lines)


The last bit handles its use on the command line. We get the file_name as the first command line argument (technically the second, the actual first argument is the name of the program that's running). We then loop through the rest of the arguments, inserting each of them into the file provided.

def main():
import sys
file_name = sys.argv[1]
for movie_id in sys.argv[2:]:
insert(movie_id, file_name)

if __name__ == '__main__':
import sys
print 'Welcome.'
print 'Inserting data for {}'.format(str(sys.argv[2:]))
main()
print 'Done.'


The point of this exercise is to show that seemingly very complex tasks can be handled easily using a combination of tools. Since TeX is a text-based format, it is very easy to generate and manipulate in countless programming languages. I hope this provides a compelling argument for the continued growth of TeX systems. :)

Any further development of this tool will be carried out on Github.

Complete, compilable code:

\begin{filecontents*}{makefile}
IDS=tt0134273 tt0050083 tt0057181
TEMPLATE=imdb-template.tex

grab:
./imdb.py $(TEMPLATE)$(IDS)

tex:
pdflatex $(TEMPLATE) pdflatex$(TEMPLATE)

all: grab tex
\end{filecontents*}
\begin{filecontents*}{imdb.py}
#!/usr/bin/env python

def get_middle(s, start, end):
import re
return re.search(re.escape(start) + '(.*?)' + re.escape(end), s).group(1)

def make_url(movie_id):
return 'http://www.imdb.com/title/{mid}'.format(
mid=movie_id)

def get_source(movie_id):
import urllib2
url = make_url(movie_id)
stream = urllib2.urlopen(url)

def get_rating(source):
# YYYY-MM-DD
iso = get_middle(source, '<span itemprop="ratingValue">', '</span>')
return iso.split('-')[0] # YYYY

def get_title(source):
return get_middle(source, '<span class="itemprop" itemprop="name">','</span>')

def get_release(source):
return get_middle(source, '<meta itemprop="datePublished" content="','" />')

def insert(movie_id, file_name, marker='%! FILMDATA !%\n'):
"""Insert information for this movie-id before the %! FILMDATA !% line in file_name.

This function should not delete the marker line.
"""
with open(file_name, 'r') as f:
file_lines = list(f)

marker_line = file_lines.index(marker)

source = get_source(movie_id)
data = '\\film{%s}{%s}{%s}{%s}\n' % (
make_url(movie_id),
get_title(source),
get_release(source),
get_rating(source))

file_lines.insert(marker_line, data)

with open(file_name, 'w') as f:
f.writelines(file_lines)

def main():
import sys
file_name = sys.argv[1]
for movie_id in sys.argv[2:]:
insert(movie_id, file_name)

if __name__ == '__main__':
import sys
print 'Welcome.'
print 'Inserting data for {}'.format(str(sys.argv[2:]))
main()
print 'Done.'
\end{filecontents*}
\documentclass[
a4paper,
12pt
]{article}

\usepackage[
hmargin=2.4cm,
vmargin=3cm
]{geometry}
\usepackage{amssymb}
\usepackage{enumitem}
\usepackage{totcount}
\usepackage[
urlcolor=black
]{hyperref}

\newcommand*\film[4]{%
\item[\refstepcounter{enumi}\textcolor{blue}{$\maltese$} \textsf{[#4]}] \textsf{\href{#1}{#2}~(#3)}}

\regtotcounter{enumi}

\begin{document}
\begin{enumerate}[leftmargin=5em,labelindent=-5em]
%! FILMDATA !%
\end{enumerate}

I have \total{enumi}~films.
\end{document}

• If you want to add more types of data, go onto an IMDB page, right click and select 'View Source' (or something similar). Search for the information you want to include; you'll probably find a couple instances of what you're looking for, but try to find something that is wrapped in explicit tags (not inline with other text). Then just use get_middle to grab that piece of information and stick it into your macro. – Sean Allred Nov 8 '13 at 18:35
• This works great, Sean. Thank you very much for the nice script. – Svend Tveskæg Nov 11 '13 at 10:57
• It was fun! I'm glad it works for you. – Sean Allred Nov 11 '13 at 14:14
• Assume I only want the year and not month and day, too; how do I get this? (If you would change the complete, compilable code, I would be very glad.) – Svend Tveskæg May 23 '15 at 15:57
• @SvendTveskæg See edit :) – Sean Allred May 23 '15 at 16:15

LuaTeX is ideal for such tasks. LuaTeX has a socket library, that can be used to fetch data from urls. In ConTeXt, almost all file related commands (\readfile, \typefile, \externalimage, etc) can fetch data from URLs as well.

However, for getting IMDB ratings, it is easier to use an API rather than parse data from the HTML page. One unofficial API from IMDB data is http://www.omdbapi.com/.

The implementation below uses the movie titles to search the IMDB. The returned value is a JSON string. For example, when you search for 12 Angry Men, the returned value is (I reformatted it, so that it is easy to read)

{
"Title":"12 Angry Men",
"Year":"1957",
"Rated":"NOT RATED",
"Released":"1 Apr 1957",
"Runtime":"96 min",
"Genre":"Drama",
"Director":"Sidney Lumet",
"Writer":"Reginald Rose (story)",
"Actors":"Martin Balsam, John Fiedler, Lee J. Cobb, E.G. Marshall",
"Plot":"A dissenting juror in a murder trial slowly manages to convince the others that the case is not as obviously clear as it seemed in court.",
"Language":"English",
"Country":"USA",
"Awards":"Nominated for 3 Oscars. Another 18 wins & 6 nominations.",
"Poster":"http://ia.media-imdb.com/images/M/MV5BODQwOTc5MDM2N15BMl5BanBnXkFtZTcwODQxNTEzNA@@._V1_SX300.jpg",
"Metascore":"N/A",
"imdbRating":"8.9",
"imdbID":"tt0050083",
"Type":"movie",
"Response":"True"
}


I then use the json module to convert it to a Lua table, which can then be easily queried for the relevant information. For example, the Lua table corresponding to the above json string is (note that the keys have been sorted alphabetically):

{
["Actors"]="Martin Balsam, John Fiedler, Lee J. Cobb, E.G. Marshall",
["Awards"]="Nominated for 3 Oscars. Another 18 wins & 6 nominations.",
["Country"]="USA",
["Director"]="Sidney Lumet",
["Genre"]="Drama",
["Language"]="English",
["Metascore"]="N/A",
["Plot"]="A dissenting juror in a murder trial slowly manages to convince the others that the case is not as obviously clear as it seemed in court.",
["Poster"]="http://ia.media-imdb.com/images/M/MV5BODQwOTc5MDM2N15BMl5BanBnXkFtZTcwODQxNTEzNA@@._V1_SX300.jpg",
["Rated"]="NOT RATED",
["Released"]="1 Apr 1957",
["Response"]="True",
["Runtime"]="96 min",
["Title"]="12 Angry Men",
["Type"]="movie",
["Writer"]="Reginald Rose (story)",
["Year"]="1957",
["imdbID"]="tt0050083",
["imdbRating"]="8.9",
}


Each individual value can be easily accessed using the lua syntax table[key]. Here is the complete code:

\usemodule[json]
\startluacode
local tolua = utilities.json.tolua
local fetch = resolvers.schemes.fetchstring
local format = string.format
local escape = url.escape

thirddata = thirdata or {}
function thirddata.imdbinfo(title)
local url    = format("http://www.omdbapi.com/?t=%s", escape(title))
local info   = fetch(url)
local parsed = tolua(info)
local value  = ""

if parsed then
value = format("[%s] %s (%s)",
parsed["imdbRating"],
parsed["Title"],
parsed["Year"])
else
end
context(value)
end
\stopluacode

\define[1]\getFilm{\ctxlua{thirddata.imdbinfo("#1")}}

\starttext
\startlines
\getFilm{8MM}
\getFilm{12 Angry Men}
\stoplines
\stoptext


which gives

I have not bother to get the correct formatting, which can be added easily.

In principle, the above code should also work with LuaLaTeX, but I don't know if there are LuaLaTeX equivalent to the l-url.lua and utl-json.lua libraries of ConTeXt.

Edit: With a slight change in the imdbinfo function, you can get posters as well.

\startluacode
local tolua = utilities.json.tolua
local fetch = resolvers.schemes.fetchstring
local format = string.format
local escape = url.escape

thirddata = thirdata or {}
function thirddata.imdbinfo(title)
local url    = format("http://www.omdbapi.com/?t=%s", escape(title))
local info   = fetch(url)
local parsed = tolua(info)
local value  = ""
local poster = ""

if parsed then
value = format("[%s] %s (%s)",
parsed["imdbRating"],
parsed["Title"],
parsed["Year"])
poster = format("\\externalfigure[%s][width=0.3\\textwidth]",
parsed["Poster"])
else
end
context.startcontent()
context(poster)
context.stopcontent()
context.startcaption()
context(value)
context.stopcaption()
end
\stopluacode

\define[1]\getFilm{\ctxlua{thirddata.imdbinfo("#1")}}

\starttext
\startcombination[2]
\getFilm{8MM}
\getFilm{12 Angry Men}
\stopcombination

\stoptext


which gives

Since I directly use \externalfigure[url], the poster is downloaded once and then the results are cached for 1 day.

• This is truly a phenomenal application of LuaTeX :) Nice find! – Sean Allred May 21 '15 at 17:12
• Very nice, Aditya! Upvoted immediately. – Svend Tveskæg May 23 '15 at 15:58
• @SvendTveskæg Besides, using an API for this shunts off some responsibility onto someone else :) As you can see, my answer wasn't nearly so clever. – Sean Allred May 23 '15 at 16:11