automated indexing against keyword file

Question

I use the package pdfpages to include scanned and ocr'd pdf's in my LaTeX document, and I need to index them (over 200 pages). I have a plain .txt file with all keywords that I'd like to find my ocr'd pdf, and those keywords that are present in my ocr'd pdf should then be indexed.

What would be the best way to do this?

Answer 1

As others said, this cannot lead to good index, because not every term usage is important, this cannot find concepts, there will be no cross references, you cannot deal with synonymy, homonyms etc.

But if you really want, there is simple script in lua, autoindex.lua:

#!/usr/bin/env texlua
local indexterms = arg[1]
local file = io.open(indexterms,"r")
if not file then 
    print("Cannot load index terms: ".. inputfile)
    os.exit()
end
local terms = {}
for line in file:lines() do
    terms[#terms + 1] = line
end
file:close()
local text = io.read("*all")
local page = 1 
local words = {}
-- Process pages
for t in text:gmatch("[^\f]*") do
    --tokenize words
    --add more characters which can't be part of words
    for x in t:gmatch("([^%s%.,!%?%(%)%-i@%$]+)") do
        local x = string.lower(x) -- normalize strings. note that this doesn't handle unicode
        local w = words[x] or {}
        w[page] = true
        words[x] = w
    end
    page = page + 1
end
for _, term in pairs(terms) do
    local match = words[term] or {}
    for page,_ in pairs(match) do
        print('\indexentry{'..term..'}{'..page..'}')
    end
end

You must first convert your pdf file to text with pdftotext utility:

pdftotext filename.pdf outputfile.txt

this will preserve page breaks. Then call this script like:

texlua autoindex.lua filewithterms < outputfile.txt > indexfile.idx

This will write entries in standard makeindex format to indexfile.idx:

indexentry{hello}{13}
indexentry{hello}{9}
indexentry{hello}{7}
indexentry{world}{7}
indexentry{world}{3}
indexentry{world}{13}
indexentry{world}{9}
indexentry{world}{5}

you can make index with makeindex or xindy then.