Programatically style words/sentences/paragraphs based on arbitrary metrics?

It'd be nice to be able to style words or sentences or subsections based on arbitrary metrics. For example, it might be useful to highlight words based on their length - bolder highlighting for longer words, so that you can zoom out of the document, and get a quick picture of where it is getting complex. Another example along the same lines might be to highlight paragraphs based on their readability score.

For the last example, this might involve something like converting the paragraph to plain text, calculating the score (externally?), mapping the score to a colour, then applying the color to the paragraph.

Is it possible to do something like this on a document-wide basis? ie. not having to add a command around every paragraph?

Can the length of a word/sentence/paragraph be easily calculated within tex, or should this be done externally?

-
Mere curiosity: To what kind of document/task are you planning to apply that scheme? Do your documents include mainly plain text or are there fancy things like math and own commands? — And sorry I can't help with your question, I definitely do not have the skill for things that complex. My guess would be: it's quite difficult, given that we still can't manage accurate word counts… – ienissei Aug 2 '12 at 7:08
@ienissei: I basically want to use this as tool as described in tex.stackexchange.com/questions/64716/… – naught101 Aug 2 '12 at 7:19
Then, my suggestion would be to write your thesis in Markdown (or better: Multi-Markdown), which uses mainly one-character tags (so it doesn't interfere with word count and such). Use a text editor that can handle your tasks (such things exist for sure), and then convert to LaTeX when you are all set. If you are on a Mac OS, Scrivener can do the conversion part (not free, but a great software). But, from experience, good writing is largely underrated by most algorithms, and bad writing is seldom detected. – ienissei Aug 2 '12 at 15:05
While I wouldn't put it past LuaTeX to do some interesting things with word-length and so forth, it is past me. I'd use external tools. I used this one-liner to sort plain tex(t) files into unique words when I wanted to get an idea of what things might need indexing: perl -ne 'print join("\n", split(/ /,$_));print("\n")' mytexfile.tex | sort -u > uniquewordlist.txt. Using tools like sed and awk you can do a lot of cool manipulations (though I have to look everything up every time I want to do something). Writing in markdown, as suggested, will eliminate a lot of the TeX markup too. – jon Aug 2 '12 at 18:43 1 Answer I made some small modifications to code that egreg posted here and came up with the following: • It looks alright, but it does not know how to hyphenate. I have no idea how to make it hyphenate or even if that is possible. Thus, there are two options: (1) split words anywhere to make it tidy, or (2) don't break anything and have stuff hanging into the margins. Both are included below and indicated in the comments. • There is the option to print each letter in its shaded box. However, that seems to darken everything up and make the highlighting less obvious. • You can change the highlighting to be darker or lighter by putting different (integer) values into \int_set:Nn \g_shade_weight_int {6} • You don't have to put each paragraph in its own block, however you do have to throw blocks of text into a macro. • It explodes when math is encountered. • If anything is broken, I likely won't be able to fix it. As mentioned, this is mostly a copy/paste solution. • The method was to: split blocks of text into paragraphs, the resulting paragraphs into words, measure the length of each word and set a shade based on the length, use egregs solution to box each letter of the word in the corresponding shade. Code: \documentclass{article} \usepackage{xparse} \usepackage{xcolor} \ExplSyntaxOn \makeatletter \int_new:N \g_shade_weight_int % sets the shade weight, bigger number for darker boxes, with this set to 6, a word % with 16-17 letters will be black. \int_set:Nn \g_shade_weight_int {6} \int_new:N \l_length_int \int_new:N \l_shade_int \tl_new:N \l_current_color_tl % code stolen from egreg with some small additions http://tex.stackexchange.com/a/57860/14100 \cs_new:Npn \show_boxes:n #1 { \int_set:Nn \l_length_int {\tl_count_tokens:n {#1}} \int_set:Nn \l_shade_int {\int_min:nn {\g_shade_weight_int*\l_length_int}{100}} \tl_set:No \l_current_color_tl {black!\int_use:N \l_shade_int} \begingroup\fboxrule=0pt\fboxsep=0pt \show_boxes_aux:nn#1\show_boxes_aux:nn\@empty \endgroup } \cs_new:Npn \show_boxes_aux:nn #1#2 { % The first part of the conditional deals with the last letter of each word, % and the \else deals with the remaining letters. Switching to the commented % fbox commands, will print each letter in the corresponding box. % Currently, this breaks words at any point, deleting "\hskip 0pt" will leave % all words unbroken and produce words extending into the margins. \ifx#2\show_boxes_aux:nn %\fbox{\colorbox{\tl_use:N \l_current_color_tl}{#1}}\expandafter\@gobble \fbox{\colorbox{\tl_use:N \l_current_color_tl}{\textcolor{\tl_use:N \l_current_color_tl}{#1}}}\expandafter\@gobble \else \setbox0=\hbox{\vbox{{#1}{#2}}}\setbox2=\hbox{\vbox{#1#2}}% \dimen0=\wd0 \advance\dimen0 -\wd2 %\fbox{\colorbox{\tl_use:N \l_current_color_tl}{#1}}\hskip 0pt\kern-\dimen0 \fbox{\colorbox{\tl_use:N \l_current_color_tl}{\textcolor{\tl_use:N \l_current_color_tl}{#1}}}\hskip 0pt\kern-\dimen0 \expandafter\show_boxes_aux:nn \fi#2 } \NewDocumentCommand {\colorwordsaux} { > { \SplitList { ~ } } m } { \tl_map_inline:nn {#1} { \show_boxes:n {##1}\ } } \NewDocumentCommand {\colorwords} {>{ \SplitList {\par}} +m} {\tl_map_inline:nn {#1} {\colorwordsaux {##1}\par}} \makeatother \ExplSyntaxOff \begin{document} \colorwords{ Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas a nulla augue. Nam a nisi eu elit consectetur cursus. Mauris mi ante, vulputate vel porttitor vitae, ultricies ac purus. Aliquam lacinia lacus a nisl elementum sit amet vehicula augue condimentum. Proin at turpis turpis, at facilisis leo. Integer convallis ligula eleifend quam sollicitudin consequat. Cras eget aliquet libero. Nam nulla lorem, tincidunt sed consequat eu, convallis sed quam. Aenean tincidunt lorem a dui vestibulum nec pharetra odio faucibus. Integer nec odio a sem faucibus rutrum. Maecenas vel lorem nisi, ut viverra est. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas a nulla augue. Nam a nisi eu elit consectetur cursus. Mauris mi ante, vulputate vel porttitor vitae, ultricies ac purus. Aliquam lacinia lacus a nisl elementum sit amet vehicula augue condimentum. Proin at turpis turpis, at facilisis leo. Integer convallis ligula eleifend quam sollicitudin consequat. Cras eget aliquet libero. Nam nulla lorem, tincidunt sed consequat eu, convallis sed quam. Aenean tincidunt lorem a dui vestibulum nec pharetra odio faucibus. Integer nec odio a sem faucibus rutrum. Maecenas vel lorem nisi, ut viverra est. } \end{document}  The output below doesn't correspond to the above lipsum, the first chunk below is a portion of a Dr. Seuss poem (The cat in the hat) and the second chunk is the abstract of an unintelligible theoretical physics paper on ArXiv. Edit: Here's a method that will color paragraphs, based on the Gunning Fog Index. As counting syllables didn't seem to be very easy, I fudged it a bit. Counting sentences via periods is also naive, so this is far from perfect, however, the results correspond reasonably well to those given by some online calculators. \documentclass{article} \usepackage{xparse} \usepackage{mdframed} \usepackage{xcolor} \ExplSyntaxOn \int_new:N \l_letter_count_int \int_new:N \l_word_count_int \int_new:N \l_sentence_count_int \int_new:N \l_complex_count_eight_int \int_new:N \l_complex_count_nine_int \seq_new:N \l_sentences_seq \seq_new:N \l_words_seq \fp_new:N \l_avg_sent_length_fp \fp_new:N \l_complex_count_fp \fp_new:N \l_percent_comp_fp \fp_new:N \l_block_score_fp \fp_new:N \l_shade_level_fp \NewDocumentCommand { \colorparagraph } { m } { \int_zero:N \l_letter_count_int \int_zero:N \l_word_count_int \int_zero:N \l_sentence_count_int \int_zero:N \l_complex_count_nine_int \int_zero:N \l_complex_count_eight_int %split paragraph at periods and count sentences, this doesn't account for %in sentence periods, e.g., e.g. %Changing the first arg from {.} to {.~} requires a space after the period but %then misses things like (...blah.) Then... \seq_set_split:Nnn \l_sentences_seq {.} {#1} \seq_remove_all:Nn \l_sentences_seq {} \int_set:Nn \l_sentence_count_int {\seq_count:N \l_sentences_seq} %split paragraph at spaces and count words. \seq_set_split:Nnn \l_words_seq {~} {#1} \seq_remove_all:Nn \l_words_seq {} \int_set:Nn \l_word_count_int {\seq_count:N \l_words_seq} %calculate average sentence length \fp_set:Nn \l_avg_sent_length_fp {\l_word_count_int/\l_sentence_count_int} %complex words = 3+ syllables...I fudged this. \seq_map_inline:Nn \l_words_seq { \int_compare:nT {\tl_count:n {##1} >= 9} {\int_incr:N \l_complex_count_nine_int} \int_compare:nT {\tl_count:n {##1} >= 8} {\int_incr:N \l_complex_count_eight_int} } % this is a fudge to estimate the number of 3+ syllable words \fp_set:Nn \l_complex_count_fp {(\l_complex_count_nine_int+\l_complex_count_eight_int)/2} % find the % of words that are "complex" \fp_set:Nn \l_percent_comp_fp {\l_complex_count_fp/\l_word_count_int} % apply fog index formula \fp_set:Nn \l_block_score_fp {.4*(\l_avg_sent_length_fp+100*\l_percent_comp_fp)} % assume that 35 is the maximum likely fog index, and calculate what percent of this % number the current paragraph is. \fp_set:Nn \l_shade_level_fp {\l_block_score_fp/35*100} \begin{mdframed} [hidealllines=true, % the percentage of black = percent of "max" fog index. backgroundcolor=black!\fp_to_int:N \l_shade_level_fp, innerleftmargin=3pt, innerrightmargin=3pt, leftmargin=-3pt, rightmargin=-3pt] #1 \end{mdframed} % uncomment below to display fog index below each paragraph %\fp_use:N \l_block_score_fp } \NewDocumentCommand {\colorparagraphs} {>{ \SplitList {\par}} +m} {\tl_map_inline:nn {#1} {\colorparagraph {##1}\par}} \ExplSyntaxOff \begin{document} %text from http://www.impact-information.com/impactinfo/newsletter/plwork08.htm \colorparagraphs{ Cardiac resynchronization therapy signals a new era in device-based solutions for this condition and 750,000 of the estimated five million Americans with heart failure could potentially benefit from it. Typically a late manifestation of other cardiovascular diseases, including coronary artery disease, hypertension and valvular disease, heart failure is responsible for more hospitalizations than all forms of cancer combined. As the only major cardiac disorder increasing in prevalence, it is estimated that 550,000 cases of heart failure are diagnosed each year. Approximately \$40 billion is spent to manage the condition in the United States each year.
Here is the same information written at the 7th-grade level:

New pacemakers like this regulate the heart beat. They can help 750,000 of the five million Americans who suffer from heart failure. Heart failure is a growing problem. It results from high blood pressure and disease in the arteries and valves of the heart. This year, 550,000 people suffered heart failure. More people went to the hospital with heart failure than all forms of cancer combined. In the U.S., we spend about \\$40 billion a year on it.
}
\end{document}


The first piece of text is written at the post graduate level, and the second at the 7th grade level. The fog indexes calculated for these are ~18.5 and ~7 respectively. The image below is from some other text that I tested. The fog indexes for the blocks are 18, 7, 23 and 11 respectively.

-
 With a small glitch (I'm not sure what causes it) I can get a better syllable count using the SOUL package. – Scott H. Aug 3 '12 at 21:47 Thanks Scott, that's awesome, especially the last example. But you have to wrap the whole document in \colorparagraphs{}, which doesn't work well when you're already using a ton of other packages, like bibliography tools, glossaries, etc. Is there any way to automatically apply it to each individual paragraph? – naught101 Aug 17 '12 at 1:46 Possibly, is there any particular way that you're marking paragraphs, e.g. with \paragraph or something like that? Or are they just paragraphs? Is there a particular documentclass that you're using? – Scott H. Aug 17 '12 at 5:43 Also, is there any specific way that you're ending paragraphs in your .tex file? Empty line between paragraphs, \par at the end etc. – Scott H. Aug 17 '12 at 6:02 I'm using the book class, with a single blank line to demarcate paragraphs. – naught101 Aug 18 '12 at 12:16
show 1 more comment