Is there an automatic process to create index creation?

Question

Certainly, there must be some way to somewhat create a list of words suggested to be included in an index.

Answer 1

I suggest you look at the script make-index.py (and related files) in the scripts folder of the download page at the Stacks Project (http://www.math.columbia.edu/algebraic_geometry/stacks-git/). The index it generates isn't really ideal, but at least their strategy will give you some idea how to get started. They seem to take the approach that (in a gigantic math textbook) the things which most deserve to be in the index are the italicized word(s) or phrase(s) in each definition environment. In my experience using math books, the most common reason I look something up in the index is to learn its definition, so this seems appropriate, although maybe not for books in other subjects. However you might be able to use the Stacks Project script as a guide to automate the creation of an index which suits your own needs, even if they are very different.

Answer 2

As others have mentioned, trying to automate this task would be close to impossible. But if you want to get some very rough hints of words for yourself, this is something I would try (note, requires some scripting):

Use detex or something to strip the TeX markup and then write a small script that counts the number of time each word has been used in the document. The top words in the list will probably be useless words like a, the, is, etc. But, after those, you might be able to find a few promising words.

Answer 3

In addition to what Juan A. Navarro suggested, I'd say that words which occur in chapter and section titles are likely candidates for indexing. E.g., if section 2.3 is entitled "The Virasoro Algebra", then that's probably a sufficiently important topic that other occurrences of "Virasoro algebra" should be indexed. You could write a script (in your favourite scripting language) to pull out the arguments to \section commands and the like, throw out the prepositions and articles and sort the remainder by frequency. How your script will know that the words Virasoro and algebra go together . . . well, either you call import skynet and live with the consequences, or you do some manual work with its output.

Other things to check could include words which are capitalized when not at the beginning of a sentence and words set in emphatic type.

Answer 4

I just wrote a quick python script to extract the most common words in some tex files. It uses detex to strip tex commands from the files, strips characters like ".", ",", ";", "?", "!" from the end of words, ignores words that contain # or =, ignores case and the 100 most common english words (copied from http://www.duboislc.org/EducationWatch/First100Words.html)

#!/usr/bin/python

import subprocess, glob, operator

# Tweak output here:
charsToStripFromEnd = ".,;?!"
nonWordChars = "=#"
minOccurrence = 30
skipWords = 'the of and a to in is you that it he was for on are as with his they I at be this have from or one had by word but not what all were we when your can said there use an each which she do how their if will up other about out many then them these so some her would make like him into time has look two more write go see number no way could people my than first water been call who oil its now find long down day did get come made may part e.g i.e'.split()

output = subprocess.check_output( ['detex'] + glob.glob('*.tex') )
wordList = output.split()
words = {}

for w in wordList:
  w = w.rstrip(charsToStripFromEnd).lower()
  if len(w) <= 2: continue

  isARealWord = True
  for c in nonWordChars: 
    if c in w: 
      isARealWord = True
  if not isARealWord: continue
  if w in skipWords: continue

  if not w in words:
    words[w] = 1
  else:
    words[w] += 1

sorted = sorted(words.iteritems(), key=operator.itemgetter(1))
sorted.reverse()

for item in sorted:
  print item[0], item[1]
  if item[1] < minOccurrence: break

Answer 5

As a rough hack I sometimes use the following lines to get all my definitions in bold+italic, and put them in the index.

\newcommand{\eb}[1]{{\itshape\bfseries#1}{\index{#1}}}

\renewcommand{\emph}{\eb}

Answer 6

I have just developed a set of tools to help me prepare the index for my PhD thesis. I believe this is a superior solution to the ones I have seen around here, despite its simplicity. We extract keywords using RAKE (we also support NLTK and YAKE, but they didn't perform as well), and then we provide an interactive approach to review each concept along with one or more matching strings to index (i.e., that should point to that concept). This results in a CSV file that can be manually edited. We then generate a new copy of the LaTeX project, which you can validate before overwriting your work.

The code is available at FEUP InfoLab's GitHub: https://github.com/feup-infolab/latex-index-tools

It should, at the very least, help you reduce the necessary time to prepare the index. Hope this helps!

Answer 7

You could use the glossaries package to suggest terms and acronyms for inclusion in an automatically generated glossary. It won't pick out words for inclusion on its own though, that might require a rather advanced level of natural language processing to accomplish.

Answer 8

An alternative would be to use detex and pipe that into a frequency-analyzing script:

This will show a list of the word distribution (case-sensitive), with the most frequent words first:

detex input.tex | tr -d '[:punct:]' | tr -d '[:digit:]' | tr ' ' '\n' | sort | uniq -c | sort -rn | less

This will output words that are not in the English dictionary:

detex input.tex | tr -d '[:punct:]' | tr -d '[:digit:]' | tr ' ' '\n' | sort -u | while read i; do if [ -z "`grep -i "$i" /usr/share/dict/words`" ]; then echo "$i"; fi; done

It's pretty slow; there's got to be a faster way, though.