Is there really no better solution to "Overfull \hbox" than adding a line break?

Question

I am going through a rather long book to resolve the Overfull \hbox problem. Here is an example:

varies greatly from organization to organization, team to team, and
manager to manager. Innovation is exceptional only because so many <<<<this too long
organizations fail to act on and to capture the benefit of good ideas.

The word "many" can not be hyphenated, and it reports the line is 0.77357pt too wide -- such a small amount. I really just want it to wrap this to the next line - the word is not a big one, and the additional white space would be trivial. The documentation (https://texfaq.org/FAQ-overfull) says to add a line feed, like this:

varies greatly from organization to organization, team to team, and
manager to manager. Innovation is exceptional only because so \linebreak
many organizations fail to act on and to capture the benefit of good ideas.

By putting a hard line break in there, it wraps it down. But later, if I edit the paragraph in front of that work, it may no longer need the line break, but the hard line break will cause it to do so anyway. If the paragraph changes so that this is no longer on the very right edge of the line, then I will not want a linebreak at that point. What do I do?

Isn't there a way to say "wrap this word down if it will cause an overflow, but include it normally otherwise" ????

The word "many" is not really a long word. I don't mind specially marking individual words in particular situations, but using \linebreak permanently causes a line break there, whether I need it or not. I would like it to do a line break ONLY if there is an overflow situation.

Another solution is to increase \pretolerance value. How do I know what that value is now? I need to know this so that I can set it to a slightly higher value. Is there a way to increase it by 10%. A word like "many" is not an exceptionally long work, and I have lots and lots of these problems. I would like it to automatically handle these cases.

UPDATE

Here is a MWE showing the problem:

\documentclass[paper=6in:9in,10pt,twoside,pagesize=pdftex,openright,headings=twolinechapter,chapterprefix=true]{scrbook}
\usepackage{graphicx}
\usepackage{wrapfig}
\usepackage{xcolor}
\usepackage{makeidx}
\makeindex
\voffset=0in
\topmargin = 0in
\headheight = 12pt
\headsep = 25pt
\textheight = 7in
\footskip = 30pt
\hoffset=0in
\marginparwidth = 0pt
\marginparsep = 0pt
\textwidth = 4.5in
\oddsidemargin = 0pt
\marginparpush = 0pt
\usepackage[inner=0.75in,outer=0.75in,top=1.0in, bottom=1.0in, footnotesep=.4in]{geometry}
\usepackage{fontspec}
\fontspec{Alegreya Sans SC Light}
\fontspec{Palatino Linotype}
\setmainfont[Ligatures={Common,TeX}, Numbers={OldStyle}]{Palatino Linotype}
\linespread{1.05}         % Palatino needs more leading (space between lines)
\addtokomafont{footnote}{\footnotesize\sffamily}
\deffootnote{1.5em}{1em}{% modified example from page 83
  \makebox[1.5em][l]{\textsuperscript{\thefootnotemark}}}
\renewcommand*\chapterheadstartvskip{\vspace*{.1\textheight}}
\renewcommand*\chapterheadendvskip{\vspace*{.1\textheight}}
\addtokomafont{chapter}{\fontspec{Alegreya Sans SC Light}}
\addtokomafont{chapterprefix}{\fontspec{Alegreya Sans SC Light}\Huge}
\addtokomafont{chapterprefix}{\raggedleft}
\renewcommand*{\chapterformat}{%
   \mbox{\scalebox{1}{\chapappifchapterprefix{\nobreakspace}}%
   \scalebox{4}{\color{gray}\thechapter}\enskip}}
\pdfpagewidth=\paperwidth
\pdfpageheight=\paperheight
\begin{document}
\frontmatter
\title{Innovation Management Challenge}
\author{Keith D Swenson, Jim Farris}
\date{\today}
\maketitle
\tableofcontents
\mainmatter
\chapter{Innovation Management Challenge}
\section{Innovators}
\begin{itemize}
\item Innovation Everywhere - Surprisingly, innovation is both exceptional and commonplace.  Good ideas are the product of intelligence, and all people at all levels are intelligent.  The ability to act on good ideas varies greatly from organization to organization, team to team, and manager to manager.   Innovation is exceptional only because so many organizations fail to act on and to capture the benefit of good ideas.
\end{itemize}
\end{document}

Searching for overfull \hbox recommendations, most of the examples are about hyphenating words. If it was a hyphenation problem, I could manually put the \- in the desired place and force a hyphenation. This example is unusual because the word many is only 4 letters and would never be split. It is short. The whole word needs to go to the next line. For this reason \hyphenate{} and \- can not do anything.

I tried using \sloppy which works (meaning the word is wrapped down and the paragraph in question looks fine), but created a lot of disgust among the onlookers saying that this simply 'should not be done'.

There was a suggestion to try \linebreak[3] which apparently means this is a kind of linebreak that does not always break, but the rule behind exactly when and why it breaks are not clear. Does it only break if the work following it is sticking a fraction of a point into the margin? I did not try it because is not clear at all the this will break when I want it to, and not break when the word fits.

I tried \usepackage{microtype} and it had no effect ... the word many remained at the end of the line and overflowing the hbox.

I tried \setlength{\emergencystretch}{2pt} and this works fine. Apparently emergency stretch is 0 by default. When an overfull situation is encountered, if emergency stretch is larger than 0, then it relaxes the requirements on that line, and allows the wrapping down of a slightly larger word. This is probably the best solution, but I still am curious why a small word like "many" needs such special handling in the first place.

I was told about \begin{sloppypar} and \end{sloppypar} which you put around the paragraph so that the entire document is no subject to the sloppy formatting. That is at least a way to do point solutions when nothing else seems to be working.

I got a long reply about changing the way that the printable area is determined. Good advice, but does not address the fundamental question of why a short word like "many" is not being wrapped down to the next line, when it extends .78 points into the margin.

Seems to me that the wrapping algorithm is simply wrong. Some say it may be a font problem, but does anyone know how to prove this? How do I get the current values of the settings that drive the word wrapping algorithm, in order to show that a critical value is too small (smaller than the length of the word "many")

Answer 1

There are many better solutions. Some short sugestions for a start.

1. More characters in a line is one of the simplest solutions. You should have 60 characters in a line as the starting point. More than, say, 70 can deeply improve the appearance of a paragraph.

2. If you can tolerate occasional too long spaces, \sloppy solves most of the problems. \sloppy means \tolerance 9999 \emergencystretch 3em\relax, hence really ugly paragraphs could be accepted. Hence it is rather for local usage, e.g. to a fixed paragraph.

3. If you can use pdftex, not the original TeX engine, package microtype is a very good solution.

Answer 2

This is an answer to the specific issue raised in your updated question i.e. now we have an MWE.

First, do you want to use KOMA or not? To my mind, if you do not generally agree with the design principles laid out in chapter 2 of the manual - or if you are not permitted to abide by them - then you would be better advised to choose another class. These principles are KOMA's raison d'être, and I honestly don't see why somebody would use the class if they did not want to lay their document out more-or-less according to those principles. This is not, of course, to say that it cannot be done. Clearly it can be done. Rather, I think the results will be inferior and the process of producing your document will be significantly more painful.

Second, your preamble uses three competing (and, therefore, conflicting) methods to specify and calculate page dimensions:

• typearea, loaded by scrbook
• manual specifications with \setlength
• geometry

Pick one. Obviously, this is related to the first point since, if you use scrbook, you cannot but load typearea. So if you prefer one of the others, I think you should reconsider your choice of class.

Third, your inner and outer margins are equal. This is unusual in a two-sided document because, when the book is opened, the white space in the centre will be twice as wide as that on the left and right of the text block. However, it might be that the extra width is required for the binding. I've assumed that is the case here.

The following code approximates the 3/4" horizontal margin specification. Two specifications are responsible for this. One is the value of BCOR which sets the binding offset to one half of 3/4". The other is the value of DIV which arranges things so that the outer margins are 3/4" and the inner are equal to the binding offset. This gives a total inner margin of 3/4" when the offset is included. (That is, the horizontal layout should be roughly the same as yours.)

typearea then calculates the remaining page dimensions according to the formula explained in chapter 2. There are variations on this - see the documentation for details.

However, the change of leading undermines the effectiveness of these calculations. In particular, it may result in a non-integer number of lines for the height of the text block. For this reason, KOMA is asked to redo the calculations after the leading is changed.

I've assumed that you want American hyphenation patterns based on your spelling. If not, substitute british for american. I've used babel here but you might prefer polyglossia since you are using XeTeX. (The addition of babel was not needed to eliminate the overfull box in this case but it will give you better results in general.)

\documentclass[paper=6in:9in,BCOR=9.5mm,DIV=14,10pt,twoside,pagesize=pdftex,openright,headings=twolinechapter,chapterprefix=true,american]{scrbook}
\usepackage{babel}
\usepackage{xcolor}
\usepackage{fontspec}
\setmainfont[Ligatures={Common,TeX}, Numbers={OldStyle}]{Palatino Linotype}
\linespread{1.05}%  Palatino needs more leading (space between lines)
\KOMAoptions{DIV=last}% make sure that an integer number of lines fits in the text block (p 35); this is necessary because the leading has been changed
\addtokomafont{footnote}{\footnotesize\sffamily}
\deffootnote{1.5em}{1em}{% modified example from page 83
  \makebox[1.5em][l]{\textsuperscript{\thefootnotemark}}}
\renewcommand*\chapterheadstartvskip{\vspace*{.1\textheight}}
\renewcommand*\chapterheadendvskip{\vspace*{.1\textheight}}
\addtokomafont{chapter}{\fontspec{Alegreya Sans SC Light}}
\addtokomafont{chapterprefix}{\fontspec{Alegreya Sans SC Light}\Huge}
\addtokomafont{chapterprefix}{\raggedleft}
\renewcommand*{\chapterformat}{%
  \mbox{\scalebox{1}{\chapappifchapterprefix{\nobreakspace}}%
    \scalebox{4}{\color{gray}\thechapter}\enskip}}
\begin{document}
  \frontmatter
  \title{Innovation Management Challenge}
  \author{Keith D Swenson, Jim Farris}
  \date{\today}
  \maketitle
  \tableofcontents
  \mainmatter
  \chapter{Innovation Management Challenge}
  \section{Innovators}
  \begin{itemize}
    \item Innovation Everywhere - Surprisingly, innovation is both exceptional and commonplace.  Good ideas are the product of intelligence, and all people at all levels are intelligent.  The ability to act on good ideas varies greatly from organization to organization, team to team, and manager to manager.   Innovation is exceptional only because so many organizations fail to act on and to capture the benefit of good ideas.
  \end{itemize}
\end{document}

With this code, I do not get any overfull boxes on compilation.

Here is a comparison of the page dimensions.

First, the dimensions obtained at the end of your original preamble:

* driver: xetex
* paper: <default>
* layout: <same size as paper>
* layoutoffset:(h,v)=(0.0pt,0.0pt)
* modes: twoside showframe
* h-part:(L,W,R)=(54.2025pt, 325.215pt, 54.2025pt)
* v-part:(T,H,B)=(72.26999pt, 505.89001pt, 72.26999pt)
* \paperwidth=433.62pt
* \paperheight=650.43pt
* \textwidth=325.215pt
* \textheight=505.89001pt
* \oddsidemargin=-18.06749pt
* \evensidemargin=-18.06749pt
* \topmargin=-37.0pt
* \headheight=12.0pt
* \headsep=25.0pt
* \topskip=10.0pt
* \footskip=30.0pt
* \marginparwidth=0.0pt
* \marginparsep=0.0pt
* \columnsep=10.0pt
* \skip\footins=28.90755pt
* \hoffset=0.0pt
* \voffset=0.0pt
* \mag=1000
* \@twocolumnfalse
* \@twosidetrue
* \@mparswitchtrue
* \@reversemarginfalse
* (1in=72.27pt=25.4mm, 1cm=28.453pt)

Now, the values obtained at the end of the revised preamble:

* \paperwidth=433.62pt
* \paperheight=650.43pt
* \textwidth=319.4635pt
* \textheight=514.00146pt
* \oddsidemargin=-16.19775pt
* \evensidemargin=-14.18573pt
* \topmargin=-60.46082pt
* \headheight=15.75005pt
* \headsep=18.90005pt
* \topskip=10.0pt
* \footskip=44.10013pt
* \marginparwidth=43.56319pt
* \marginparsep=12.8401pt
* \columnsep=10.0pt
* \skip\footins=9.0pt plus 4.0pt minus 2.0pt
* \hoffset=0.0pt
* \voffset=0.0pt
* \mag=1000
* \@twocolumnfalse
* \@twosidetrue
* \@mparswitchtrue
* \@reversemarginfalse
* (1in=72.27pt=25.4mm, 1cm=28.453pt)

Note that the text block is actually slightly narrower in this case but also slightly taller. The other changes to the vertical layout are a function of the formula KOMA uses.

I have not included microtype since loading it with the default options actually caused an overfull box in this case. Probably this is because the default values are not good for Palatino. This could be corrected for best results either manually or by checking what values are used when standard font packages are loaded for Palatino or TeX-Gyre Pagella. However, since micro-typography is quite limited with this engine anyway, it is probably not worth worrying about. (If you wanted the benefit of these features, you'd do better to use pdfTeX or LuaTeX.)

Adding 7 or so pages of Kant for testing, I still don't get any overfull boxes.

Answer 3

Here's an example based on one of your other questions:

\documentclass[paper=6in:9in,10pt,twoside,pagesize=pdftex,openright]{scrbook}
\usepackage{kantlipsum}

\begin{document}
  \kant[1-20]
\end{document}

This gives me 2 overfull boxes (for 10 pages of text). And this is a text with its fair share of bizarre Kantian terminology which is probably not the best input to TeX's hyphenation mechanism.

With the following I get no overfull complaints (and I think this is roughly the typearea you are using):

\documentclass[paper=6in:9in,DIV=12,10pt,twoside,pagesize=pdftex,openright]{scrbook}
\usepackage{kantlipsum}

\begin{document}
  \kant[1-20]
\end{document}

That is Computer Modern. Here is Palatino:

\documentclass[paper=6in:9in,DIV=12,10pt,twoside,pagesize=pdftex,openright]{scrbook}
\usepackage{kantlipsum}
\usepackage{tgpagella}

\begin{document}
  \kant[1-20]
\end{document}

Still no overfull boxes.

Adding

\usepackage{microtype}

will improve the quality of the output and helps to avoid overfull boxes, although it is not fully supported by/does not fully support XeTeX. (It is maximally supported by/supportive of pdfTeX and there is significant support for LuaTeX but it will still help and do no harm, at worst.)

Using babel with the appropriate option or polyglossia, I think, will make sure that the right hyphenation patterns are loaded and will help TeX find appropriate hyphenation points.

If you are getting a lot of these and you are not typesetting narrow columns of text, something is wrong. Since I can't find an MWE in any of your questions (the ones I checked), it is, of course, impossible to say very much. This is why MWEs are so important.