6

How do you count the number of word occurrences in your tex files? The reason I do this is to easier recognize words I use too much in a text. At the moment I use the following one-liner in the bash.

cat *.tex | sed 's/[[:space:]|[:punct:]]\+/\n/g' | sort | uniq -c | sort -n

What it does is, output all .tex-Files with cat, substitute the whitespaces and punctuations with a line-break using sed, sort the output, count the unique words and sort it again after the number output by uniq -c.

One of the problems I have with that approach is, that words that belong together but are divided by a whitespace are counted separately. So for example "New York" you get k occurrences of New and n occurrences of York, mixing with other occurrences of New and York.

EDIT: Another problem is of course, how do you recognize word inflection such as declension and conjugation? But that's probably something way out of scope of a one-liner, or does anyone have an idea how to cope with that?

EDIT2: As Hendrik and Joseph pointed out, that's not really TeX-related, but perhaps somebody finds it useful :)

4
  • 3
    I think your question isn't really TeX related, so I'm voting to close it as "off-topic". My point is somehow that a satisfactory answer wouldn't involve any TeX knowledge but rather access to a huge database. Mar 26, 2011 at 23:02
  • I'd agree that the question as posed does not really seem to be TeX-related. I'd vote to close if I were not a moderator.
    – Joseph Wright
    Mar 27, 2011 at 8:44
  • 1
    Alternatively, look at tex.stackexchange.com/questions/534 for a TeX-specific set of answers.
    – Joseph Wright
    Mar 27, 2011 at 8:44
  • Thanks for the comments, you're probably right that it's not only related to TeX but text-processing as a whole. I've thought that maybe other people had the same issue when writing their thesis in TeX, so I just asked.
    – Bruno
    Mar 27, 2011 at 12:40

1 Answer 1

1

Cannot say much about the run latex, and then use dvi2tty on the output .dvi file. This would take better care of macro expansion. I suggest therefore:

dvi2tty 00.dvi | sed  's/[[:space:]|[:punct:]|[:digit:]]\+/\n/g' | sed '/^$/d' | tr "A-Z" "a-z" | sort | uniq -c | sort -nr | sed "/ 1 /d" 

which is similar to your pipe, except that

  • It treats digits as separators.
  • It ignores spaces
  • It eliminates empty lines
  • It eliminates words that occur only once.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .