I am a XeLaTeX user and I often have to typeset english-greek documents. Packages like xgreek or polyglossia are great but in order to apply the correct hyphenation rules you have to declare the text that belongs to the secondary language. On the long run, this can become cumbersome.

In XeLaTeX I am using the XeTeXinterchartoks mechanism that allows me to change automatically the hyphenation rules and/or the font without explicitingly declaring them. The procedure involves grouping together the glyphs of a Unicode Block and then automatically applying tex commands when transitioning from a group to another one.

For those not familiar with the XeTeXinterchartoks, more info can be found at xetex-reference pages 13-14.

Lately I got interested in Lua(La)Tex. I want to know if there is a way to achieve similar results in LuaTex. As was pointed in a previous question, in LuaTeX there is not a direct analogue to XeTeXinterchartoks but it was suggested that there are other, more powerful ways to achieve the same goal.

So the questions are

  1. How can I change the hyphenation rules/font automatically according to the glyphs used without declaring them?
  2. Can someone provide a minimum working example or at least point me to some references?
  3. One of the drawbacks of XeTeXinterchartoks is that the settings are global. Is there a way in LuaTeX to set this feature (if it exists) on and off, thus providing more flexibility?
  • I just wanted to play with this, but I can't get a simple greek text typeset with LuaTeX/babel, as it switches to cmr (which has no greek encoding).
    – topskip
    Commented Jun 27, 2011 at 8:00
  • @Patrick. I don't think that the greek style of polyglossia actually use \XeTeXinterclass so you can probably use it if you define before polyglossia \newcommand\newXeTeXintercharclass[1]{}. You can also use babel and change only the hyphenrules with the hyphenrules-environment. Commented Jun 27, 2011 at 9:46

1 Answer 1


Here is a proof of concept at doing the equivalent of \XeTeXinterchartoks in luatex.

First, a style file:

    % luatexinterchartoks.sty




   {\afterassignment\dodoXeTeXcharclass\cchtwo }

   {\afterassignment\doXeTeXcharclass\cchone }


   {\newtoks\mytoks\afterassignment\XeTeXdointerchartoks\global\mytoks }

   {\afterassignment\dodoXeTeXinterchartoks\cchtwo }

   {\afterassignment\doXeTeXinterchartoks\cchone }



And a matching lua file:

% luatexinterchartoks.lua

charclasses = charclasses or {}

function setcharclass (a,b)
   charclasses[a] = b

local i = 0
while i < 65536 do
  charclasses[i]  = 0
  i = i + 1

interchartoks =  interchartoks or {}

function setinterchartoks (a,b,c)
   interchartoks[a] = interchartoks[a] or {}
   interchartoks[a][b] = c

local nc, oc
oc = 255

function do_intertoks () 
  local tok = token.get_next() 
  if tex.count['XeTeXinterchartokenstate'] == 1 then
      if tok[1] == 11 or  tok[1] == 12 then
        nc = charclasses[tok[2]] 
        newchar = tok[2]
        nc = 255
        newchar = ''
      local insert  = ''
      if interchartoks[oc] and interchartoks[oc][nc] then
          insert = interchartoks[oc][nc] 
          local newtok = tok
          if insert<100 then
            local dec = math.floor(insert / 10) + 48;
            local unit = math.floor(insert % 10) + 48;
            newtok = {
              -- \XeTeXinterchartokenstate=0 \the\toks<n> \XeTeXinterchartokenstate=1
              token.create(string.byte(' '),10),
              token.create(string.byte(' '),10),
              token.create(string.byte(' '),10),
              {tok[1], tok[2], tok[3]}}               
          tok = newtok
      oc = nc
  return tok

callback.register ('token_filter', do_intertoks)

And a test document:




\newXeTeXintercharclass \mycharclassa
\newXeTeXintercharclass \mycharclassA
\newXeTeXintercharclass \mycharclassB
\XeTeXcharclass `\a \mycharclassa
\XeTeXcharclass `\A \mycharclassA
\XeTeXcharclass `\B \mycharclassB
% between "a" and "A":
\XeTeXinterchartoks \mycharclassa \mycharclassA = {[\itshape}
\XeTeXinterchartoks \mycharclassA \mycharclassa = {\upshape]}
% between " " and "B":
\XeTeXinterchartoks 255 \mycharclassB = {\bgroup\color{blue}}
\XeTeXinterchartoks \mycharclassB 255 = {\egroup}
% between "B" and "B":
\XeTeXinterchartoks \mycharclassB \mycharclassB = {.}

\XeTeXinterchartokenstate = 1
aAa A a B aBa BB


Not very pretty, but it proves that it can be done ...

  • @Taco, thank you. I will have to check it more thoroughly in order to understand what you did. To be honest, I was hoping that it would be more straightforward but hopefully I'll figure it out.
    – pmav99
    Commented Jun 27, 2011 at 12:11
  • That looks really good. I had to replace the dofile by a require so that the lua-file is found with kpathsea. (And probably in a real .sty the callback.register should be replaced by the corresponding luatexbase.add_to_callback). And then polyglossia work fine and e.g. the french gave the correct spaces. Commented Jun 27, 2011 at 12:49
  • @Ulrike: I simply do not know all that stuff ;) Commented Jun 27, 2011 at 15:35
  • 3
    @Ulrike: the code above can be taken by anyone that wants or needs it (public domain), and I also am willing to help. But I do not want to be the maintainer of a LaTeX package. Too much to do already and in any case, it would be weird to be the maintainer of a package I neither need nor use myself. Commented Jun 27, 2011 at 17:02
  • 2
    @Taco: unfortunately, your code is not equivalent to what XeTeX does, since your code inserts the tokens between each character-token in the input stream, while XeTeX inserts them between character nodes being added to the current list. So using it for characters that may be used outside typesetting-context, as :;!? (as does polyglossia with French) is likely to break a huge lot of things. I don't think XeTeXinterchartoks can be correctly emulated in LuaTeX, but I don't think it's a big problem either, since LuaTeX provides a different (and superior) set of tools.
    – mpg
    Commented Jul 2, 2011 at 17:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .