Is there an automatic process to create index creation?

Question

Certainly, there must be some way to somewhat create a list of words suggested to be included in an index.

Answer 1

I suggest you look at the script make-index.py (and related files) in the scripts folder of the download page at the Stacks Project (http://www.math.columbia.edu/algebraic_geometry/stacks-git/). The index it generates isn't really ideal, but at least their strategy will give you some idea how to get started. They seem to take the approach that (in a gigantic math textbook) the things which most deserve to be in the index are the italicized word(s) or phrase(s) in each definition environment. In my experience using math books, the most common reason I look something up in the index is to learn its definition, so this seems appropriate, although maybe not for books in other subjects. However you might be able to use the Stacks Project script as a guide to automate the creation of an index which suits your own needs, even if they are very different.

Answer 2

As others have mentioned, trying to automate this task would be close to impossible. But if you want to get some very rough hints of words for yourself, this is something I would try (note, requires some scripting):

Use detex or something to strip the TeX markup and then write a small script that counts the number of time each word has been used in the document. The top words in the list will probably be useless words like a, the, is, etc. But, after those, you might be able to find a few promising words.

Answer 3

In addition to what Juan A. Navarro suggested, I'd say that words which occur in chapter and section titles are likely candidates for indexing. E.g., if section 2.3 is entitled "The Virasoro Algebra", then that's probably a sufficiently important topic that other occurrences of "Virasoro algebra" should be indexed. You could write a script (in your favourite scripting language) to pull out the arguments to \section commands and the like, throw out the prepositions and articles and sort the remainder by frequency. How your script will know that the words Virasoro and algebra go together . . . well, either you call import skynet and live with the consequences, or you do some manual work with its output.

Other things to check could include words which are capitalized when not at the beginning of a sentence and words set in emphatic type.

Answer 4

I just wrote a quick python script to extract the most common words in some tex files. It uses detex to strip tex commands from the files, strips characters like ".", ",", ";", "?", "!" from the end of words, ignores words that contain # or =, ignores case and the 100 most common english words (copied from http://www.duboislc.org/EducationWatch/First100Words.html)

#!/usr/bin/python

import subprocess, glob, operator

# Tweak output here:
charsToStripFromEnd = ".,;?!"
nonWordChars = "=#"
minOccurrence = 30
skipWords = 'the of and a to in is you that it he was for on are as with his they I at be this have from or one had by word but not what all were we when your can said there use an each which she do how their if will up other about out many then them these so some her would make like him into time has look two more write go see number no way could people my than first water been call who oil its now find long down day did get come made may part e.g i.e'.split()

output = subprocess.check_output( ['detex'] + glob.glob('*.tex') )
wordList = output.split()
words = {}

for w in wordList:
  w = w.rstrip(charsToStripFromEnd).lower()
  if len(w) <= 2: continue

  isARealWord = True
  for c in nonWordChars: 
    if c in w: 
      isARealWord = True
  if not isARealWord: continue
  if w in skipWords: continue

  if not w in words:
    words[w] = 1
  else:
    words[w] += 1

sorted = sorted(words.iteritems(), key=operator.itemgetter(1))
sorted.reverse()

for item in sorted:
  print item[0], item[1]
  if item[1] < minOccurrence: break

Answer 5

As a rough hack I sometimes use the following lines to get all my definitions in bold+italic, and put them in the index.

\newcommand{\eb}[1]{{\itshape\bfseries#1}{\index{#1}}}

\renewcommand{\emph}{\eb}

Answer 6

You could use the glossaries package to suggest terms and acronyms for inclusion in an automatically generated glossary. It won't pick out words for inclusion on its own though, that might require a rather advanced level of natural language processing to accomplish.