# How can I efficiently make TeX read each visible letter as a word?

In other words,

• How can I allow breaks after any visible character? ...[such that lines are broken when the sum of the lengths of horizontal boxes equals \linewidth]

• How can I disable or influence the TeX spacing/line-breaking algorithm to allow for breaks whenever the line is full?

This is a follow-up to a question asked earlier How can I make LaTeX to recognize spaces in my macro (catcode 10)?.

I ran into two problems:

1. Dealing with the active character
2. Dealing with memory

The active character causes issues (as expected). David Carlisle mentioned that I could use \noexpand to properly handle the active character. I tried, but to no avail. Furthermore, because the entire text is fed into the macro, the whole thing is stored in memory before it is actually shipped out.

How this thing works:

1. \treateachletterasword sends its entire input to a scanner (this is the memory hog)
2. The scanner, \xscan, sends a single token to \xxscan by grabbing a token with \afterassignment
3. \xxscan tests for catcodes and works accordingly
4. Finally, \xxscan calls \xscan on the next token (this loop breaks when the equivalent of \relax is encountered-may or may not be the best solution)

# Code

(David Carlisle mentioned that instead of \hskip 0pt plus 1sp minus 1sp, I should use \hskip 0pt, or even better, \penalty0 to inprove efficiency. This way, TeX does not need to calculate as much for each line.)

\documentclass{article}
\usepackage{fontspec}

\newfontfamily\monofont{Inconsolatazi4-Regular.otf}% let's pretend this is not a mono font for practice
%\makeatletter % or \catcode"0040=11
\long\def\treateachletterasword#1{\xscan #1\relax}% calls initial \xscan on first char
\def\xscan{\afterassignment\xxscan\let\token= }% assign next single token to \token and send it to \xxscan
\def\xxscan{%
\monofont% apply mono font
\ifx\token\relax\normalfont\else%test for end-of-line or end of group and switch back to normal font
\ifcat\token\space%
\token% token is catcode 10
\spaceskip=.5em% remove glue from space for fixed-width space and precise control
\xspaceskip=.5em% remove glue from space for fixed-width space and precise control
\else%
\ifcat\token\active% deal with active character backslash
%\noexpand\token
\textbackslash% for lack of a solution, this macro should be expanded
\else
\token\hskip 0pt plus 1sp minus 1sp% add glue to any non-catcode 10 (space)
\fi
\fi
\expandafter\xscan%calls subsequent xscan until next char representd by \token = \relax
\fi}
%\makeatother % or \catcode"0040=12
\parindent=0pt % remove firstpar autoindent
\obeylines% insert \par after each end-of-line (^^M)

\begin{document}
\section{SomeTeXFile.tex}\subsection{Typesetting Failure}
\treateachletterasword{
This is XeTeX, Version 3.14159265-2.6-0.99996 (TeX Live 2016) (preloaded format=xelatex 2016.6.27)  11 AUG 2017 13:15
entering extended mode
restricted \char"005C{}write18 enabled.
file:line:error style messages enabled.
\%\&-line parsing enabled.
LaTeX2e <2016/03/31> patch level 2
Babel <3.9r> and hyphenation patterns for 83 language(s) loaded.
(../../DocumentClass.tex (/usr/local/texlive/2016/texmf-dist/tex/latex/base/article.cls
Document Class: article 2014/09/29 v1.4h Standard LaTeX document class
(/usr/local/texlive/2016/texmf-dist/tex/latex/base/size10.clo
File: size10.clo 2014/09/29 v1.4h Standard LaTeX file (size option)
)
\char"005C{}c@part=\char"005C{}count79
\char"005C{}c@section=\char"005C{}count80
\char"005C{}c@subsection=\char"005C{}count81
\char"005C{}c@subsubsection=\char"005C{}count82
\char"005C{}c@paragraph=\char"005C{}count83
\char"005C{}c@subparagraph=\char"005C{}count84
\char"005C{}c@figure=\char"005C{}count85
\char"005C{}c@table=\char"005C{}count86
\char"005C{}abovecaptionskip=\char"005C{}skip41
\char"005C{}belowcaptionskip=\char"005C{}skip42
\char"005C{}bibindent=\char"005C{}dimen102
}
\end{document}


# Output

Note that

1. By adding glue to every non-catcode 10 (except active char) and removing glue from every catcode 10 (space), TeX fills lines with as many boxes as it takes to fill the line without stretching spaces. This effectively disables the TeX beautifying mechanism.
2. Macros should be expanded, but they are not. See \"005C, for example.
3. Here is another point I forgot to mention: The new lines are ignored despite \obeylines. This is because both } and \par (from ^^M) would yield \relax. This is undesired.

• What is your end goal supposed to look like? For example, does \obeylines\obeyspaces\ttfamily instead of \treateachletterasword{} suffice? Aug 11, 2017 at 12:19
• @StevenB.Segletes I added the output. And no. Aug 11, 2017 at 12:25
• what is a "letter" here? is å a letter, or ﭴ ? they are multiple tokens to pdftex (which is exactly the issue that listings has when it tries to do exactly this and space out "characters") Aug 11, 2017 at 12:35
• last question are you using luatex or xetex? (both have features that probably make this easier) lua callbacks or xetex inter-character classes respectively. (The xetex character classes were specifically designed to add inter-character glue to allow line breaking in asian languages that don't use space, which seems to be very close to this) Aug 11, 2017 at 12:42
• \ifcat\token\active does not test if the token is active, if the token is expandable it tests the first two tokens in its expansion have the same catcode, otherwise it tests if the token is not a character (as all other non expandable tokens, including \active test as equal to \ifcat) Aug 11, 2017 at 12:52

I was able to solve this with a concept unique to XeTeX: The Character and Intercharacter Classes as suggested by David Carlisle. Will Robertson and Khaled Hosny did a good job documenting this functionality here, http://mirror.unl.edu/ctan/info/xetexref/xetex-reference.pdf.

There seem to be two ways to set delimiters for the injection of characters between tokens using \XeTeXinterchartoks:

1. Between character slots and intercharacter classes. e.g. Use \XeTeXcharclass to assign \A to a \macro and finally define the injection as \XeTeXinterchartoks \macro 0 = {\penalty0\relax}. This adds a penalty of 0 after every occurance of the "A" character when it is followed by a character of class 0.
2. Between intercharacter classes. e.g. An intercharacter class is a number from 0 to 255 that represents a group of related characters. e.g. \XeTeXinterchartoks 0 0 = {\penalty0\relax}. Unfortunately, this part of the document was unclear to me, so I did a quick search and found another document that sheds more light on the intercharacter classes: https://github.com/zohooo/interchar/blob/master/interchar.tex.

# Code

I removed the token scanner in this version. It is no longer necessary, perhaps because the \XeTeXinterchartoks functionality takes place before or during the assignment of category codes. I am not sure exactly how it works. The only downside is that I lost control over spaces. There might be a way to implement this control again using character classes.

\documentclass{article}
\usepackage{fontspec}
\usepackage{lipsum}

\newfontfamily\monofont{Inconsolatazi4-Regular.otf}% let's pretend this is not a mono font for practice

\long\def\treateachletterasword{%
\bgroup
\XeTeXinterchartokenstate=1% Enable Character Classes (unique to xelatex) 0=off 1=on
\XeTeXinterchartoks 0 0 = {\penalty0\relax}% Set token to be inserted between interchar class 0 and interchar class 0
\monofont% Set monospaced font
\setlength{\parindent}{0pt}% rm new par indents
\obeylines% \catcode\^^M\active \let ^^M\par
}%
\def\endtreateachletterasword{\egroup}

\begin{document}
\section{SomeTeXFile.tex}\subsection{Typesetting Failure}
\begin{treateachletterasword}
This is XeTeX, Version 3.14159265-2.6-0.99996 (TeX Live 2016) (preloaded format=xelatex 2016.6.27)  11 AUG 2017 13:15
entering extended mode
restricted \char"005C{}write18 enabled.
file:line:error style messages enabled.
\%\&-line parsing enabled.
LaTeX2e <2016/03/31> patch level 2
Babel <3.9r> and hyphenation patterns for 83 language(s) loaded.
(../../DocumentClass.tex (/usr/local/texlive/2016/texmf-dist/tex/latex/base/article.cls
Document Class: article 2014/09/29 v1.4h Standard LaTeX document class
(/usr/local/texlive/2016/texmf-dist/tex/latex/base/size10.clo
File: size10.clo 2014/09/29 v1.4h Standard LaTeX file (size option)
)
\char"005C{}c@part=\char"005C{}count79
\char"005C{}c@section=\char"005C{}count80
\char"005C{}c@subsection=\char"005C{}count81
\char"005C{}c@subsubsection=\char"005C{}count82
\char"005C{}c@paragraph=\char"005C{}count83
\char"005C{}c@subparagraph=\char"005C{}count84
\char"005C{}c@figure=\char"005C{}count85
\char"005C{}c@table=\char"005C{}count86
\char"005C{}abovecaptionskip=\char"005C{}skip41
\char"005C{}belowcaptionskip=\char"005C{}skip42
\char"005C{}bibindent=\char"005C{}dimen102
\end{treateachletterasword}
\end{document}


# Output

Be careful! The \relax after the \penalty0 is necessary else consecutive numbers will be gobbled up by TeX as shown in the output image. I have since corrected this mistake in the code. Also, I have since added \setlength{\parindent}{0pt} to remove the new paragraph indentation after each new line of code caused by \obeylines.