In LuaTex is it possible to change font/language according to the script/glyphs used?

I am a XeLaTeX user and I often have to typeset english-greek documents. Packages like xgreek or polyglossia are great but in order to apply the correct hyphenation rules you have to declare the text that belongs to the secondary language. On the long run, this can become cumbersome.

In XeLaTeX I am using the XeTeXinterchartoks mechanism that allows me to change automatically the hyphenation rules and/or the font without explicitingly declaring them. The procedure involves grouping together the glyphs of a Unicode Block and then automatically applying tex commands when transitioning from a group to another one.

For those not familiar with the XeTeXinterchartoks, more info can be found at xetex-reference pages 13-14.

Lately I got interested in Lua(La)Tex. I want to know if there is a way to achieve similar results in LuaTex. As was pointed in a previous question, in LuaTeX there is not a direct analogue to XeTeXinterchartoks but it was suggested that there are other, more powerful ways to achieve the same goal.

So the questions are

1. How can I change the hyphenation rules/font automatically according to the glyphs used without declaring them?
2. Can someone provide a minimum working example or at least point me to some references?
3. One of the drawbacks of XeTeXinterchartoks is that the settings are global. Is there a way in LuaTeX to set this feature (if it exists) on and off, thus providing more flexibility?
-
I just wanted to play with this, but I can't get a simple greek text typeset with LuaTeX/babel, as it switches to cmr (which has no greek encoding). –  topskip Jun 27 '11 at 8:00
@Patrick. I don't think that the greek style of polyglossia actually use \XeTeXinterclass so you can probably use it if you define before polyglossia \newcommand\newXeTeXintercharclass[1]{}. You can also use babel and change only the hyphenrules with the hyphenrules-environment. –  Ulrike Fischer Jun 27 '11 at 9:46

Here is a proof of concept at doing the equivalent of \XeTeXinterchartoks in luatex.

First, a style file:

    % luatexinterchartoks.sty
\newcount\XeTeXinterchartokenstate

\newcount\charclasses
\def\newXeTeXintercharclass#1%
\newcount#1
\global#1=\the\charclasses
}

\newcount\cchone
\newcount\cchtwo

\def\dodoXeTeXcharclass
{\directlua{setcharclass(\the\cchone,\the\cchtwo)}}

\def\doXeTeXcharclass%
{\afterassignment\dodoXeTeXcharclass\cchtwo }

\def\XeTeXcharclass%
{\afterassignment\doXeTeXcharclass\cchone }

\protected\def\XeTeXdointerchartoks%
{\directlua{setinterchartoks(\the\cchone,\the\cchtwo,\the\allocationnumber)}}

\protected\def\dodoXeTeXinterchartoks%
{\newtoks\mytoks\afterassignment\XeTeXdointerchartoks\global\mytoks }

\protected\def\doXeTeXinterchartoks%
{\afterassignment\dodoXeTeXinterchartoks\cchtwo }

\def\XeTeXinterchartoks%
{\afterassignment\doXeTeXinterchartoks\cchone }

\luatexdirectlua{dofile('luatexinterchartoks.lua')}

\endinput


And a matching lua file:

% luatexinterchartoks.lua

charclasses = charclasses or {}

function setcharclass (a,b)
charclasses[a] = b
end

local i = 0
while i < 65536 do
charclasses[i]  = 0
i = i + 1
end

interchartoks =  interchartoks or {}

function setinterchartoks (a,b,c)
interchartoks[a] = interchartoks[a] or {}
interchartoks[a][b] = c
end

local nc, oc
oc = 255

function do_intertoks ()
local tok = token.get_next()
if tex.count['XeTeXinterchartokenstate'] == 1 then
if tok[1] == 11 or  tok[1] == 12 then
nc = charclasses[tok[2]]
newchar = tok[2]
else
nc = 255
newchar = ''
end
local insert  = ''
if interchartoks[oc] and interchartoks[oc][nc] then
insert = interchartoks[oc][nc]
local newtok = tok
if insert<100 then
local dec = math.floor(insert / 10) + 48;
local unit = math.floor(insert % 10) + 48;
newtok = {
-- \XeTeXinterchartokenstate=0 \the\toks<n> \XeTeXinterchartokenstate=1
token.create('XeTeXinterchartokenstate'),
token.create(string.byte('='),12),
token.create(string.byte('0'),12),
token.create(string.byte(' '),10),
token.create('the'),
token.create('toks'),
token.create(dec,12),
token.create(unit,12),
token.create(string.byte(' '),10),
token.create('XeTeXinterchartokenstate'),
token.create(string.byte('='),12),
token.create(string.byte('1'),12),
token.create(string.byte(' '),10),
{tok[1], tok[2], tok[3]}}
end
tok = newtok
end
oc = nc
end
end

callback.register ('token_filter', do_intertoks)


And a test document:

\documentclass{article}

\usepackage{luatexinterchartoks}
\usepackage{color}

\begin{document}

\newXeTeXintercharclass \mycharclassa
\newXeTeXintercharclass \mycharclassA
\newXeTeXintercharclass \mycharclassB
\XeTeXcharclass \a \mycharclassa
\XeTeXcharclass \A \mycharclassA
\XeTeXcharclass \B \mycharclassB
% between "a" and "A":
\XeTeXinterchartoks \mycharclassa \mycharclassA = {[\itshape}
\XeTeXinterchartoks \mycharclassA \mycharclassa = {\upshape]}
% between " " and "B":
\XeTeXinterchartoks 255 \mycharclassB = {\bgroup\color{blue}}
\XeTeXinterchartoks \mycharclassB 255 = {\egroup}
% between "B" and "B":
\XeTeXinterchartoks \mycharclassB \mycharclassB = {.}

\begingroup
\XeTeXinterchartokenstate = 1
aAa A a B aBa BB
\endgroup

\end{document}


Not very pretty, but it proves that it can be done ...

-
@Taco, thank you. I will have to check it more thoroughly in order to understand what you did. To be honest, I was hoping that it would be more straightforward but hopefully I'll figure it out. –  pmav99 Jun 27 '11 at 12:11
That looks really good. I had to replace the dofile by a require so that the lua-file is found with kpathsea. (And probably in a real .sty the callback.register should be replaced by the corresponding luatexbase.add_to_callback). And then polyglossia work fine and e.g. the french gave the correct spaces. –  Ulrike Fischer Jun 27 '11 at 12:49
@Ulrike: I simply do not know all that stuff ;) –  Taco Hoekwater Jun 27 '11 at 15:35
@Ulrike: the code above can be taken by anyone that wants or needs it (public domain), and I also am willing to help. But I do not want to be the maintainer of a LaTeX package. Too much to do already and in any case, it would be weird to be the maintainer of a package I neither need nor use myself. –  Taco Hoekwater Jun 27 '11 at 17:02
@Taco: unfortunately, your code is not equivalent to what XeTeX does, since your code inserts the tokens between each character-token in the input stream, while XeTeX inserts them between character nodes being added to the current list. So using it for characters that may be used outside typesetting-context, as :;!? (as does polyglossia with French) is likely to break a huge lot of things. I don't think XeTeXinterchartoks` can be correctly emulated in LuaTeX, but I don't think it's a big problem either, since LuaTeX provides a different (and superior) set of tools. –  mpg Jul 2 '11 at 17:19