Automatically adjust interletter spacing (tracking) in XeTeX

Background: We know that getting fully justified text, without resorting to hyphenation, is a hard problem and invariably requires awful spaces. The situation would be slightly improved if inter-letter spacing could also be increased and not just inter-word spacing. Here is an illustration from the soul documentation:

3.5 Dirty tricks

Narrow columns are hard to set, because they don’t allow much spacing flexibility, hence long words often cause overfull boxes. A macro could use \so to insert stretchability between the single characters. Table 2 shows…

The middle one is what I want (won't have so much letterspacing though!), in XeTeX. (Have to use XeTeX because the document requires OpenType fonts and preferably Unicode input, and LuaTeX does not support the script I need.) I'd like it to apply to the entire book, without having to surround each paragraph with \foo{...}.

Here's a document:

\documentclass{article}
\usepackage{fontspec}
\usepackage{polyglossia}

\begin{document}

\hsize=300pt
\hyphenpenalty=10000
\emergencystretch=100em

ಶ್ರೀವನಿತೆಯರಸನೆ ವಿಮಲ ರಾಜೀವ ಪೀಠನ ಪಿತನೆ ಜಗಕತಿ ಪಾವನನೆ ಸನಕಾದಿ
ಸಜ್ಜನನಿಕರ ದಾತಾರ. ರಾವಾಣಾಸುರ ಮಥನ ಶ್ರವಣ ಸುಧಾ ವಿನೂತನ ಕಥನ,
ಕಾರಣ ಕಾವುದಾನತ ಜನವ. ಗದುಗಿನ ವೀರನಾರಯಣ ಶರಣಸಂಗವ್ಯಸನ.
ಭುಜಗಾಭರಣನಮರ ಕಿರೀಟ ಮಂಡಿತಚರಣ ಚಾರುಚರಿತ್ರ. ನಿರುಪಮ ಭಾಳಶಿಖಿನೇತ್ರ
ಕರಣನಿರ್ಮಲ ಭಜಕರಘ. ಸಂಹರಣ ದಂತಿ. ಚಮೂರು ಚರ್ಮಾಂಬರನೆ
ಸಲಹುಗೆ ಭಕುತ ಜನರನು ಪಾರ್ವತೀರಮಣ. ವರಮಣಿಗಳಿಂದೆಸೆವ ಮೌಳಿಯ
ಸರಸಿಜಾರಿಯ ಕಿರಣದೋಳಿಯ ವಿರಚಿಸಿದ ಸಿಂಧೂರಭಾಳದಿ ಕುಣಿವ ಕುಂತಳದ
ಕರಿ ನಿಭಾಕೃತಿಯೆನಿಪ ವದನದ ಕರದ ಪಾಶದ ಮೋದಕದ ವಿಸ್ತರದ ಗಣಪತಿ
ಮಾಡೆಮಗೆ ನಿರ್ವಿಘ್ನದಾಯಕವ.

\end{document}


which produces

Note the wide spaces especially on line 6 of 9. I'd like to decrease these spaces, by slightly increasing the spaces between the letters on that line (letterspacing aka tracking).

In general, I'd like to increase the spaces between the letters when a line is underfull, by an amount up to some limit I consider aesthetically acceptable, but keep them at their normal values by default. (That is, stretch only when necessary, and only up to some limit.) I know people will have mixed opinions about the typographic propriety of doing this, but in this case I'm trying to match the look of an existing book and that's what the publisher seems to have done.

All the questions I could find on this site are about indiscriminately and persistently increasing letter spacing:

I have read the UK TeX FAQ on letterspacing and looked at letterspacing.tex (documented in TUGboat), at microtype and at soul, in all of which I found only “increase letterspacing globally” and not the feature I was looking for (except for the above note from the soul documentation).

• I looked at soul.dtx and it has a \magstylepar, but I can't seem to get it to work. Jul 18 '17 at 16:21
• Well, I got \magstylepar to work with English text, but it doesn't work with the text above. :-( Jul 18 '17 at 16:30
• After thinking about this with a clear head for a moment, I suspect that what I want might be quite difficult: knowing what is a “glyph” is at font level… how could TeX know where to add spaces? Jul 18 '17 at 16:41
• You could teach it, or teach me:-) by classifying the characters and saying how much space to add between each class. Jul 18 '17 at 16:42

microtype helps a bit but you can add space via character classes:

\documentclass{article}
\usepackage{fontspec}
\usepackage{polyglossia}
\XeTeXinterchartokenstate=1
\newXeTeXintercharclass\Kclass
\newXeTeXintercharclass\Kclassb
\makeatletter
\count@"00C80
\loop
\XeTeXcharclass\count@=\Kclass
\ifnum\count@<"00D00
\repeat

\XeTeXcharclass"0C81=\Kclassb
\XeTeXcharclass"0C82=\Kclassb
\XeTeXcharclass"0C83=\Kclassb
\XeTeXcharclass"0CBC=\Kclassb
\XeTeXcharclass"0CBE=\Kclassb
\XeTeXcharclass"0CBF=\Kclassb
\XeTeXcharclass"0CC0=\Kclassb
\XeTeXcharclass"0CC1=\Kclassb
\XeTeXcharclass"0CC2=\Kclassb
\XeTeXcharclass"0CC3=\Kclassb
\XeTeXcharclass"0CC4=\Kclassb
\XeTeXcharclass"0CC6=\Kclassb
\XeTeXcharclass"0CC7=\Kclassb
\XeTeXcharclass"0CC8=\Kclassb
\XeTeXcharclass"0CCA=\Kclassb
\XeTeXcharclass"0CCB=\Kclassb
\XeTeXcharclass"0CCC=\Kclassb
\XeTeXcharclass"0CCD=0 % Don't insert hskip before or after this character
\XeTeXcharclass"0CD5=\Kclassb
\XeTeXcharclass"0CD6=\Kclassb
\XeTeXcharclass"0CE2=\Kclassb
\XeTeXcharclass"0CE3=\Kclassb
\XeTeXcharclass"0CF1=\Kclassb
\XeTeXcharclass"0CF2=\Kclassb

\XeTeXinterchartoks\Kclass\Kclass{\nobreak\hskip0pt plus .5pt\relax}
\XeTeXinterchartoks\Kclassb\Kclass{\nobreak\hskip0pt plus .5pt\relax}
\makeatother

\usepackage{microtype}
\begin{document}
\XeTeXinterchartokenstate=1
\hsize=300pt
\hyphenpenalty=10000
\emergencystretch=100em

ಶ್ರೀವನಿತೆಯರಸನೆ ವಿಮಲ ರಾಜೀವ ಪೀಠನ ಪಿತನೆ ಜಗಕತಿ ಪಾವನನೆ ಸನಕಾದಿ
ಸಜ್ಜನನಿಕರ ದಾತಾರ. ರಾವಾಣಾಸುರ ಮಥನ ಶ್ರವಣ ಸುಧಾ ವಿನೂತನ ಕಥನ,
ಕಾರಣ ಕಾವುದಾನತ ಜನವ. ಗದುಗಿನ ವೀರನಾರಯಣ ಶರಣಸಂಗವ್ಯಸನ.
ಭುಜಗಾಭರಣನಮರ ಕಿರೀಟ ಮಂಡಿತಚರಣ ಚಾರುಚರಿತ್ರ. ನಿರುಪಮ ಭಾಳಶಿಖಿನೇತ್ರ
ಕರಣನಿರ್ಮಲ ಭಜಕರಘ. ಸಂಹರಣ ದಂತಿ. ಚಮೂರು ಚರ್ಮಾಂಬರನೆ
ಸಲಹುಗೆ ಭಕುತ ಜನರನು ಪಾರ್ವತೀರಮಣ. ವರಮಣಿಗಳಿಂದೆಸೆವ ಮೌಳಿಯ
ಸರಸಿಜಾರಿಯ ಕಿರಣದೋಳಿಯ ವಿರಚಿಸಿದ ಸಿಂಧೂರಭಾಳದಿ ಕುಣಿವ ಕುಂತಳದ
ಕರಿ ನಿಭಾಕೃತಿಯೆನಿಪ ವದನದ ಕರದ ಪಾಶದ ಮೋದಕದ ವಿಸ್ತರದ ಗಣಪತಿ
ಮಾಡೆಮಗೆ ನಿರ್ವಿಘ್ನದಾಯಕವ.

\XeTeXinterchartokenstate=0

ಶ್ರೀವನಿತೆಯರಸನೆ ವಿಮಲ ರಾಜೀವ ಪೀಠನ ಪಿತನೆ ಜಗಕತಿ ಪಾವನನೆ ಸನಕಾದಿ
ಸಜ್ಜನನಿಕರ ದಾತಾರ. ರಾವಾಣಾಸುರ ಮಥನ ಶ್ರವಣ ಸುಧಾ ವಿನೂತನ ಕಥನ,
ಕಾರಣ ಕಾವುದಾನತ ಜನವ. ಗದುಗಿನ ವೀರನಾರಯಣ ಶರಣಸಂಗವ್ಯಸನ.
ಭುಜಗಾಭರಣನಮರ ಕಿರೀಟ ಮಂಡಿತಚರಣ ಚಾರುಚರಿತ್ರ. ನಿರುಪಮ ಭಾಳಶಿಖಿನೇತ್ರ
ಕರಣನಿರ್ಮಲ ಭಜಕರಘ. ಸಂಹರಣ ದಂತಿ. ಚಮೂರು ಚರ್ಮಾಂಬರನೆ
ಸಲಹುಗೆ ಭಕುತ ಜನರನು ಪಾರ್ವತೀರಮಣ. ವರಮಣಿಗಳಿಂದೆಸೆವ ಮೌಳಿಯ
ಸರಸಿಜಾರಿಯ ಕಿರಣದೋಳಿಯ ವಿರಚಿಸಿದ ಸಿಂಧೂರಭಾಳದಿ ಕುಣಿವ ಕುಂತಳದ
ಕರಿ ನಿಭಾಕೃತಿಯೆನಿಪ ವದನದ ಕರದ ಪಾಶದ ಮೋದಕದ ವಿಸ್ತರದ ಗಣಪತಿ
ಮಾಡೆಮಗೆ ನಿರ್ವಿಘ್ನದಾಯಕವ.

\end{document}


Here I have given the whole range the class Kclass, but then re-assigned combining characters to class Kclassb so that space is not inserted between the base and the combining character. It is hopefully more readable than the first version that only used one class, but as I can't read it, it is hard to be sure. More classes could be used if some characters could have bigger spaces than others.

• @ShreevatsaR Yes as I say this breaks combining characters but not doing that is just a matter of not making combining characters have Kclass so replacing that simple loop by something that actually has some knowledge about the characters in that range Jul 18 '17 at 16:44
• Ooh interesting idea, let me explore it and see if it can work! Jul 18 '17 at 16:45
• @ShreevatsaR there are no dotted circles now, but honestly I do not know if it is any more readable. Jul 18 '17 at 18:08
• it is clear that I have broken some combinations still, as the letter forms in the top version differ from those in the bottom, hopefully more, finer, classification can address that. Jul 18 '17 at 18:35
• Very clever, this works as far as I can tell! I was trying this myself and ended up with a lot more classes (which became useful as a sort of linting for unexpected input). The main refinement needed is to put say \XeTeXcharclass"0CCD = 0, because that character behaves like neither of two classes above. Really impressive :-) Jul 18 '17 at 20:14