This might be interesting for perfectionists and/or fastidious typesetters who would like to improve the document even further (beyond the beyond the magnificence of a book with zero bad boxes).
We all know that the hyphenation algorithm, as conceived by Mr. Franklin Mark Liang and implemented in the patgen programme, is based on the processing of a whole bunch of pre-hyphenated words, calculating the likelihood of a permitted break, building a compact table for the sake of space and memory efficiency etc. It may correctly identify up to 90% possible breaks depending on the language. However, given that the computers of the present day are no longer constrained by the limitations of ’82, the hyphenation in *TeX output can be further improved in two ways:
- We can create a comprehensive hyphenation database for each language and get the 100% accuracy.
- We can deal with the problem if it appears: analyse the logged report, find any occurrences of overfulls and, if their roots stretch back to undiscovered allowed hyphenation point, manually add the word to the “white list” of
\hyphenation{...}.
While in English language this doesn’t seem to be the problem owing to the abundance of short words, in Russian and German it is a frequent case: sometimes I get 5 overfull boxes only due to the missed hyphenation point, and after some hard-coding similar to \hyphenation{ми-н-да-лём ра-с-по-ря-ди-те-лю мо-ж-но}, all the bad boxes are gone. As a typesetter of Russian texts, I can assure you that they look perfect after TeX with [russian]{babel} that beautifully handles all the diverse punctuation, but the practice of tying (~) one-letter words (and ideally some two-letter auxiliaries) to the following word is begloomed by some unfound hyphenation points, and overfulls ensue. Since many-many books are compact in size, the text area is often limited to 100×175 mm, or even smaller. Believe me, this is a real challenge for a typesetter of cyrillic texts.
Problem in one sentence: a missed hyphenation point causes some inferior line breaking or one close to æsthetically unacceptable, and after manual hyphenation is introduced, a new breakpoint is used, improving the breaking; however, manual “leak plugging” is unfunny.
Any additional non-breakable space (~) is a restriction, and we all know that mathematically, it cannot decrease the general “badness” of a paragraph. Any additional condition is a compromise that causes the conditional minimum obtained through the minimisation of the cubic function to grow in comparison to what it would have been if the restriction had not been imposed (same for regression analysis: the restricted sum of squared residuals is greater than the unrestricted one). The problem is aggravated by the fact that TeX does not report if the badness of a line does not exceed 1000 but comes very close to it.
It would be much nicer if I could track down and hunt, say, a line with badness 990 that is only that bad because a breakpoint of a word was not found by the hyphenation algorithm! It would be much nicer to see all the spots where the breakpoints were missed and add more degrees of freedom, thus improving the look (if a word has to be broken anyway, better give it a maximum number of breaks allowed by the rules of the language!).
This has driven me to the point of two possible ways of dealing with the problem:
- Make and compile a DIY modification of pdfLaTeX that would report every occurence of
\badnessexceeding X (say 700) in every line in which a word had to be hyphenated, which is undoubtedly a dirty hack; - Write an extension that would display the badness after each line (kind of “über-
draft” mode that not only prints a black rectangle where the overflow has occurred, but rather report instances of the interword space being close to its maximum or minimum allowed value.
I thought that it might be possible in LuaTeX that the absolute amount of glue added to the standard interword space (3.33333pt plus 1.66666pt minus 1.11111pt, if I am correct) were printed in the margins. If it is possible in LuaTeX, then it can be pushed further to being user-friendly: the percentage of possible amount shrunk or expanded printed... and coloured (it’s LuaTeX, after all!). UPDATE: But obviously it uses different fonts and metrics, and such a solution would not help any of LaTeX typesetters, who, as I roughly estimate, make up a large majority of TeX users, and the proportion is not likely to waver.
If microtype package is enabled, the same question arises: can we obtain the stretch/shrink parameter values for each line of output? If the default limit is 20, then a value of 20 or −19 in a line in which a word break occurs may indicate that it could not find a hyphenation point and therefore had to resort to extreme expansion/compression.
Although the transition to full-size hyphenation dictionaries may be the most beautiful option in the long run (assuming that the complexity of the hyphenation search algorithm does not exceed... say, O(n·log(n)), where n is the measure of the dictionary size), all I want for now is the ascertainment of the possibility of printing/storing the badness of each line and/or the exact amount of glue added/removed.
Desired result in one sentence: review occurrences of
\badnesses close to critical or amount of glue added close to maximum allowed (the concept is shown in the figures).
(This is an approximate model of what could become the new quality criterion for LaTeX output.)
What can you advise?
UPDATE
I have reproduced a bothering example in which a manual \hyphenation of a word drastically improved the paragraph layout.
Minimal working example:
\documentclass[10pt]{memoir}
\usepackage[T2A]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[russian]{babel} % Enable Russian hyphenation
\usepackage{microtype} % See how even microtype fails
\righthyphenmin=2 % Russian language rules
\def\psk{\hskip1.5em\relax} % Parboxes and all that hard-coded stuff just pursue
% the illustrative aim to reproduce the example precisely
\begin{document}
\parbox[t]{226.15pt}{\psk И~он показывал какую-то странную позу, несколько
запрокинувшись назад, как бы полупадая от «истомлённости».}
% The badness is very close to 1000, and you see how bad it is
\parbox[t]{226.2pt}{\psk И~он показывал какую-то странную позу, несколько
запрокинувшись назад, как бы полупадая от «истомлённости».}
% Now the badness is over 1000
\parbox[t]{226.15pt}{\psk И~он показывал какую-то странную позу, не\-с\-ко\-ль\-ко
запрокинувшись назад, как бы полупадая от «истомлённости».}
% Since there must a hyphen anyway, this breaking is much more beautiful now!
% (And such hyphenation is perfectly legitimate.)
\end{document}
Underfull \hbox (badness 1009) in paragraph at lines 15--15
[] \T2A/cmr/m/n/10 (+20) И он по-ка-зы-вал какую-то стран-ную по-зу,
This is what I was talking about: there must be one hyphen in a paragraph, and both layouts 1 and 3 are not reported as bad, but the manually adjusted 3 is more beautiful. Of course one can run the document multiple times with \textwidth ranging, for instance, from 220 to 250 pt with step 5, and manually amend all those ugly lines by providing all possible breakpoints, but... You know... LaTeX documents are not meant to be improved by some hard-coding, r-right?
Nota bene: if \parboxes are used, then unfound hyphenation causes underfulls. If the same width is passed as a parameter to the geometry package and the text is typeset as normal paragraphs, unfound hyphenation causes overfulls. Both are odious, though.
P.S. I am aware of the http://tug.org/TUGboat/tb31-3/tb99isambert.pdf article by Mr. Paul Isambert that introduces a Lua(La)TeX way to look at the page grey evenness. Besides, the chickenize package provides the \colorstretch function that blindly evaluates everything. However, I do not deem highly of Lua(La)TeX’s robustness and stability (with respect to input) since there are so many things to manually detect and recode with some hand-kludged typography tools (thin spaces, thin nbsp’s, initial spaces—holy cow, there is no way to dispose of good old babel) in Unicode in place of nice and decent LaTeX macros! Just to illustrate that there is a not-so-robust solution which may or may not be reimplemented if LaTeX, please see the following example (polyglossia's hyphenation goes smash, too):
\documentclass[10pt,oneside]{memoir}
\usepackage{fontspec}
\usepackage{polyglossia}
\usepackage{microtype} % See how even microtype fails
\righthyphenmin=2 % Russian language rules
\setmainfont{Liberation Serif}
\setdefaultlanguage{russian}
\setlength{\parindent}{1.5em}
\usepackage[textwidth=200.2pt]{geometry}
\usepackage{chickenize}
\begin{document}
\colorstretch
И~он показывал какую-то странную позу, несколько
запрокинувшись назад, как бы полупадая от «истомлённости».
И~он показывал какую-то странную позу, не\-с\-ко\-ль\-ко
запрокинувшись назад, как бы полупадая от «истомлённости».
\end{document}
(Compiled on Linux Mint Debian without any additional fonts installed.) Well, this does not reproduce the exact spacing and goodness of the handcrafted paragraph, but gives a slight idea of what I desire to see in LaTeX—a mean of detection of possible inferior breaking that was caused by a missed hyphenation point.
\hyphenation{...}either. you also don't mention one proposal that has been made before -- enhancing the hyphenation pattern mechanism to distinguish between compounds (also to some extent prefixes and suffixes) and "ordinary" hyphenation points. otherwise an interesting idea. (have dealt with russian before when setting a russian-english math dictionary. set it ragged right.).affformat files are something similar to this idea; whereas WYSIWYG editors directly benefit from those, in case of TeX, they have to undergo some reforge before they form a basis for syntax parsing.\hbadness=700at the start of your document to report all lines with badness >700 instead of >1000. For displaying the badness of each line separately, it might be possible to abuse thelinenopackage, which already manages to split the document into individual lines and place a number in front: to find the badness of a given line, set it in a box of size\hsize(or\linewidth?) and look at the value of\badness.