Take the 2-minute tour ×
TeX - LaTeX Stack Exchange is a question and answer site for users of TeX, LaTeX, ConTeXt, and related typesetting systems. It's 100% free, no registration required.

I am a XeLaTeX user and I often have to typeset english-greek documents. Packages like xgreek or polyglossia are great but in order to apply the correct hyphenation rules you have to declare the text that belongs to the secondary language. On the long run, this can become cumbersome.

In XeLaTeX I am using the XeTeXinterchartoks mechanism that allows me to change automatically the hyphenation rules and/or the font without explicitingly declaring them. The procedure involves grouping together the glyphs of a Unicode Block and then automatically applying tex commands when transitioning from a group to another one.

For those not familiar with the XeTeXinterchartoks, more info can be found at xetex-reference pages 13-14.

Lately I got interested in Lua(La)Tex. I want to know if there is a way to achieve similar results in LuaTex. As was pointed in a previous question, in LuaTeX there is not a direct analogue to XeTeXinterchartoks but it was suggested that there are other, more powerful ways to achieve the same goal.

So the questions are

  1. How can I change the hyphenation rules/font automatically according to the glyphs used without declaring them?
  2. Can someone provide a minimum working example or at least point me to some references?
  3. One of the drawbacks of XeTeXinterchartoks is that the settings are global. Is there a way in LuaTeX to set this feature (if it exists) on and off, thus providing more flexibility?
share|improve this question
    
I just wanted to play with this, but I can't get a simple greek text typeset with LuaTeX/babel, as it switches to cmr (which has no greek encoding). –  topskip Jun 27 '11 at 8:00
    
@Patrick. I don't think that the greek style of polyglossia actually use \XeTeXinterclass so you can probably use it if you define before polyglossia \newcommand\newXeTeXintercharclass[1]{}. You can also use babel and change only the hyphenrules with the hyphenrules-environment. –  Ulrike Fischer Jun 27 '11 at 9:46
add comment

1 Answer

up vote 11 down vote accepted

Here is a proof of concept at doing the equivalent of \XeTeXinterchartoks in luatex.

First, a style file:

    % luatexinterchartoks.sty
\newcount\XeTeXinterchartokenstate

\newcount\charclasses
\def\newXeTeXintercharclass#1%
  {\global\advance\charclasses1\relax
   \newcount#1
   \global#1=\the\charclasses
   }

\newcount\cchone
\newcount\cchtwo

\def\dodoXeTeXcharclass
    {\directlua{setcharclass(\the\cchone,\the\cchtwo)}}

\def\doXeTeXcharclass%
   {\afterassignment\dodoXeTeXcharclass\cchtwo }

\def\XeTeXcharclass%
   {\afterassignment\doXeTeXcharclass\cchone }

\protected\def\XeTeXdointerchartoks%
   {\directlua{setinterchartoks(\the\cchone,\the\cchtwo,\the\allocationnumber)}}

\protected\def\dodoXeTeXinterchartoks%
   {\newtoks\mytoks\afterassignment\XeTeXdointerchartoks\global\mytoks }

\protected\def\doXeTeXinterchartoks%
   {\afterassignment\dodoXeTeXinterchartoks\cchtwo }

\def\XeTeXinterchartoks%
   {\afterassignment\doXeTeXinterchartoks\cchone }

\luatexdirectlua{dofile('luatexinterchartoks.lua')}

\endinput

And a matching lua file:

% luatexinterchartoks.lua

charclasses = charclasses or {}

function setcharclass (a,b)
   charclasses[a] = b
end

local i = 0
while i < 65536 do
  charclasses[i]  = 0
  i = i + 1
end

interchartoks =  interchartoks or {}

function setinterchartoks (a,b,c)
   interchartoks[a] = interchartoks[a] or {}
   interchartoks[a][b] = c
end

local nc, oc
oc = 255

function do_intertoks () 
  local tok = token.get_next() 
  if tex.count['XeTeXinterchartokenstate'] == 1 then
      if tok[1] == 11 or  tok[1] == 12 then
        nc = charclasses[tok[2]] 
        newchar = tok[2]
      else 
        nc = 255
        newchar = ''
      end
      local insert  = ''
      if interchartoks[oc] and interchartoks[oc][nc] then
          insert = interchartoks[oc][nc] 
          local newtok = tok
          if insert<100 then
            local dec = math.floor(insert / 10) + 48;
            local unit = math.floor(insert % 10) + 48;
            newtok = {
              -- \XeTeXinterchartokenstate=0 \the\toks<n> \XeTeXinterchartokenstate=1
              token.create('XeTeXinterchartokenstate'),
              token.create(string.byte('='),12),
              token.create(string.byte('0'),12),
              token.create(string.byte(' '),10),
              token.create('the'),
              token.create('toks'),
              token.create(dec,12),
              token.create(unit,12),
              token.create(string.byte(' '),10),
              token.create('XeTeXinterchartokenstate'),
              token.create(string.byte('='),12),
              token.create(string.byte('1'),12),
              token.create(string.byte(' '),10),
              {tok[1], tok[2], tok[3]}}               
          end
          tok = newtok
      end
      oc = nc
  end
  return tok
end

callback.register ('token_filter', do_intertoks)

And a test document:

\documentclass{article}

\usepackage{luatexinterchartoks}
\usepackage{color}

\begin{document}

\newXeTeXintercharclass \mycharclassa
\newXeTeXintercharclass \mycharclassA
\newXeTeXintercharclass \mycharclassB
\XeTeXcharclass `\a \mycharclassa
\XeTeXcharclass `\A \mycharclassA
\XeTeXcharclass `\B \mycharclassB
% between "a" and "A":
\XeTeXinterchartoks \mycharclassa \mycharclassA = {[\itshape}
\XeTeXinterchartoks \mycharclassA \mycharclassa = {\upshape]}
% between " " and "B":
\XeTeXinterchartoks 255 \mycharclassB = {\bgroup\color{blue}}
\XeTeXinterchartoks \mycharclassB 255 = {\egroup}
% between "B" and "B":
\XeTeXinterchartoks \mycharclassB \mycharclassB = {.}

\begingroup
\XeTeXinterchartokenstate = 1
aAa A a B aBa BB
\endgroup

\end{document}

Not very pretty, but it proves that it can be done ...

share|improve this answer
    
@Taco, thank you. I will have to check it more thoroughly in order to understand what you did. To be honest, I was hoping that it would be more straightforward but hopefully I'll figure it out. –  pmav99 Jun 27 '11 at 12:11
    
That looks really good. I had to replace the dofile by a require so that the lua-file is found with kpathsea. (And probably in a real .sty the callback.register should be replaced by the corresponding luatexbase.add_to_callback). And then polyglossia work fine and e.g. the french gave the correct spaces. –  Ulrike Fischer Jun 27 '11 at 12:49
    
@Ulrike: I simply do not know all that stuff ;) –  Taco Hoekwater Jun 27 '11 at 15:35
2  
@Ulrike: the code above can be taken by anyone that wants or needs it (public domain), and I also am willing to help. But I do not want to be the maintainer of a LaTeX package. Too much to do already and in any case, it would be weird to be the maintainer of a package I neither need nor use myself. –  Taco Hoekwater Jun 27 '11 at 17:02
1  
@Taco: unfortunately, your code is not equivalent to what XeTeX does, since your code inserts the tokens between each character-token in the input stream, while XeTeX inserts them between character nodes being added to the current list. So using it for characters that may be used outside typesetting-context, as :;!? (as does polyglossia with French) is likely to break a huge lot of things. I don't think XeTeXinterchartoks can be correctly emulated in LuaTeX, but I don't think it's a big problem either, since LuaTeX provides a different (and superior) set of tools. –  mpg Jul 2 '11 at 17:19
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.