6

I have tried to use the new xindex indexing package on a Norwegian manuscript. I modified the xindex-cfg-common.lua as indicated in the documentation, and made a character matrix with some minor changes. Adding Norwegian to the folium{} also works, which is invoked by -l no.

The new configuration file is named xindex-norsk.lua, is loaded with the commands:

\usepackage[imakeidx]{xindex}
\makeindex[options=-c norsk -l no]

It works to some extent, but it will not sort ÆØÅæøå in correct order, and will not recognize that æøå is the lower case of ÆØÅ.

I have tested code from other configuration files without any luck, since I have no knowledge of Lua (or any other programming language). For example, the sorting routine from the xindex-yannis.lua resulted in no output at all when I copied it to my xindex-norsk.lua file.

The output when using my xindex-norsk.lua is like this:

enter image description here

The upper-/lowercase issue can be ‘fixed’ by forcing sort order

åkerhøne\index{Åkerhøne@åkerhøne}

but the wrong sorting of the character means lot of manual editing of the .idx-file.

Is there any easy way to fix this?

Complete .cfg and MWE:

-----------------------------------------------------------------------
--         FILE:  xindex-norsk.lua
--  DESCRIPTION:  configuration file for xindex.lua
-- REQUIREMENTS:  
--       AUTHOR:  Herbert Voß
--     MODIFIED:  Sveinung Heggen (2020-01-02)
--      LICENSE:  LPPL1.3
-----------------------------------------------------------------------

if not modules then modules = { } end modules ['xindex-cfg'] = {
      version = 0.20,
      comment = "configuration to xindex.lua",
       author = "Herbert Voss",
    copyright = "Herbert Voss",
      license = "LPPL 1.3"
}

escape_chars = { -- by default " is the escape char
  {'""', "\\escapedquote",      '\"{}' },
  {'"@', "\\escapedat",         "@"    },
  {'"|', "\\escapedvert",       "|"    },
  {'"!', "\\escapedexcl",       "!"    },
  {'"(', "\\escapedparenleft",  "("   },
  {'")', "\\escapedparenright", ")"  }
}

itemPageDelimiter = ","     -- Hello, 14
compressPages     = true    -- something like 12--15, instead of 12,13,14,15. the |( ... |) syntax is still valid
fCompress         = true    -- 3f -> page 3, 4 and 3ff -> page 3, 4, 5
minCompress       = 3       -- 14--17 or 
numericPage       = true    -- for non-numerical page numbers, like "VI-17"
sublabels         = {"", "---\\,", "--\\,", "-\\,"} -- for the (sub(sub(sub-items  first one is for item
pageNoPrefixDel   = ""     -- a delimiter for page numbers like "VI-17"
indexOpening      = ""     -- commands after \begin{theindex}
rangeSymbol       = "--"
idxnewletter      = "\\textbf"  -- Only valid if -n is not set

folium = { 
  de = {"f.", "ff."},
  en = {"f.", "ff."},
  fr = {"\\,sq","\\,sqq"},
  no = {"\\,f.","\\,ff."},
} 


--[[
    Each character's position in this array-like table determines its 'priority'.
    Several characters in the same slot have the same 'priority'.
]]

alphabet_lower = { --   for sorting
    { ' ' },  -- only for internal tests
    { 'a', 'á', 'à', },
    { 'b' },
    { 'c', 'ç' },
    { 'd' },
    { 'e', 'é', 'è', 'ë', 'ê' },
    { 'f' },
    { 'g' },
    { 'h' },
    { 'i', 'í', 'ì', 'î', 'ï' },
    { 'j' },
    { 'k' },
    { 'l' },
    { 'm' },
    { 'n', 'ñ' },
    { 'o', 'ó', 'ò', 'ô' },
    { 'p' },
    { 'q' },
    { 'r' },
    { 's', 'š', 'ß' },
    { 't' },
    { 'u', 'ú', 'ù', 'û' },
    { 'v' },
    { 'w' },
    { 'x' },
    { 'y', 'ý', 'ÿ', 'ü' },
    { 'z', 'ž' },
    { 'æ', 'œ', 'ä' },
    { 'ø', 'ö' },
    { 'å' }
}
alphabet_upper = { -- for sorting
    { ' ' },
    { 'A', 'Á', 'À', 'Â'},
    { 'B' },
    { 'C', 'Ç' },
    { 'D' },
    { 'E', 'È', 'É', 'Ë', 'Ê' },
    { 'F' },
    { 'G' },
    { 'H' },
    { 'I', 'Í', 'Ì', 'Ï', 'Î' },
    { 'J' },
    { 'K' },
    { 'L' },
    { 'M' },
    { 'N', 'Ñ' },
    { 'O', 'Ó', 'Ò', 'Ô' },
    { 'P' },
    { 'Q' },
    { 'R' },
    { 'S', 'Š' },
    { 'T' },
    { 'U', 'Ú', 'Ù', 'Û' },
    { 'V' },
    { 'W' },
    { 'X' },
    { 'Y', 'Ý', 'Ÿ', 'Ü' },
    { 'Z', 'Ž' },
    { 'Æ', 'Œ', 'Ä' },
    { 'Ø', 'Ö' },
    { 'Å' }
}

Test file for indexed words (MWE):

\documentclass[norsk]{scrartcl}
\usepackage{babel}

\usepackage[imakeidx]{xindex}
\makeindex[options=-c norsk -l no]

\begin{document}

Anemone\index{Anemone|(}
Anemone\index{Anemone}
Anemone\index{Anemone}

Àctomere\index{Àctomere}
Bartëloemeus\index{Bartëloemeus}
Bartelomeus\index{Bartelomeus}
Cicero\index{Cicero}
Citroën\index{Citroën}
Delta\index{Delta}
Epsilon\index{Epsilon}
Forfatter\index{Forfatter}

For\index{For}
Fôr\index{Fôr}
Får\index{Får}
Färdsel\index{Färdsel}
Færøyene\index{Færøyene}
Anemone\index{Anemone}
Anemone\index{Anemone}

Gavepakker\index{Gavepakker}
Grisgrendt\index{Grisgrendt}
Göteborg\index{Göteborg}
Heggen\index{Heggen}
Havørn\index{Havørn}
Havtor\index{Havtor}
Indigo\index{Indigo}
Înderer\index{Înderer}
Interesse\index{Interesse}
Joker\index{Joker}
Jôkum\index{Jôkum}
Jökullen\index{Jökullen}
København\index{København}
Kjære\index{Kjære}
Kjøre\index{Kjøre}
Kjole\index{Kjole}
Kåre\index{Kåre}
Lam\index{Lam}
Lammeull\index{Lammeull}
Mammakjole\index{Mammakjole}
Mamelukk\index{Mamelukk}
Mehe\index{Mehe}
Noen\index{Noen}
Neon\index{Neon}
Oppetid\index{Oppetid}
Oppenheim\index{Oppenheim}
Ôskardir\index{Ôskardir}
Pappenheimer\index{Pappenheimer}
Prada\index{Prada}
Praha\index{Praha}
Quisling\index{Quisling}
Quest\index{Quest}
Raske\index{Raske}
Raska\index{Raska}
Ramböll\index{Ramböll}
Sistemann\index{Sistemann}
Sitroner\index{Sitroner}
Utenpå\index{Utenpå}
Überall\index{Überall}
Varig\index{Varig}
Verdipapir\index{Verdipapir}
Wikipedia\index{Wikipedia}
Xindy\index{Xindy}
Xindex\index{Xindex}
Xanadu\index{Xanadu}
Ytterplagg\index{Ytterplagg}
Ynkelig\index{Ynkelig}
Zackariassen\index{Zackariassen}
Zambia\index{Zambia}
Ærfugl\index{Ærfugl}
Ærekjær\index{Ærekjær}
Øredøvende\index{Øredøvende}
Øvelse\index{Øvelse}
Äckligt\index{Äckligt}
Åkerhøne\index{Åkerhøne}
Åkerhane\index{Åkerhane}

\newpage
Anemone\index{Anemone}
Anemone\index{Anemone|)}

anemone\index{anemone}
àctomere\index{àctomere}
bartëloemeus\index{bartëloemeus}
bartelomeus\index{bartelomeus}
cicero\index{cicero}
citroën\index{citroën}
delta\index{delta}
epsilon\index{epsilon}
forfatter\index{forfatter}
fôr\index{fôr}
for\index{for}
får\index{får}
färdsel\index{färdsel}
færøyene\index{færøyene}
gavepakker\index{gavepakker}
grisgrendt\index{grisgrendt}
göteborger\index{göteborger}
heggen\index{heggen}
havørn\index{havørn}
havtor\index{havtor}
indigo\index{indigo}
înderer\index{înderer}
interesse\index{interesse}
joker\index{joker}
jôkum\index{jôkum}
jökullen\index{jökullen}
københavn\index{københavn}
kjære\index{kjære}
kjøre\index{kjøre}
kjole\index{kjole}
lam\index{lam}
lammeull\index{lammeull}
mammakjole\index{mammakjole}
mamelukk\index{mamelukk}
mehe\index{mehe}
noen\index{noen}
neon\index{neon}
oppetid\index{oppetid}
oppenheim\index{oppenheim}
ôskardir\index{ôskardir}
pappenheimer\index{pappenheimer}
prada\index{prada}
praha\index{praha}
quisling\index{quisling}
quest\index{quest}
raske\index{raske}
raska\index{raska}
ramböll\index{ramböll}
sistemann\index{sistemann}
sitroner\index{sitroner}
utenpå\index{utenpå}
überall\index{überall}
varig\index{varig}
verdipapir\index{verdipapir}
wikipedia\index{wikipedia}
xindy\index{xindy}
xindex\index{xindex}
xanadu\index{xanadu}
ytterplagg\index{ytterplagg}
ynkelig\index{ynkelig}
zackariassen\index{zackariassen}
zambia\index{zambia}
ærfugl\index{ærfugl}
ærekjær\index{ærekjær}
øredøvende\index{øredøvende}
øvelse\index{øvelse}
äckligt\index{äckligt}
åkerhøne\index{Åkerhøne@åkerhøne}
åkerhane\index{åkerhane}


\printindex

\end{document}
  • Maybe you could try the dirty trick I used long ago for my PHD tex.stackexchange.com/questions/272156/… – sztruks Jan 5 at 15:08
  • 1
    @sztruks I have investigated that, i could search and replace all Æ or æ with a sorting key Zz1, all Ø or ø with Zz2 etc., i.e (\index{Zz1verby@Øverby}) etc. Then you can use makeindx directly. But it was this type of hacking I hoped to avoid by using xindex. – Sveinung Jan 5 at 16:32
  • Is it the normal behaviour that ÆÅ are not sorted under A and Ø not under O? – user187802 Jan 17 at 7:45
  • @user187802 ÆØÅ is independent letters as is the last three letters in the Norwegian alphabet and is always sorted in that order. We have 29 letters. The sorting order is ÆæØøÅå (capital letters first). Umlauts with the same sound are normally sorted together: Ä -> Æ, Ö -> Ø and Ü -> Y. They are not ordinary letter in Norwegian, but are used in names, brands etc. – Sveinung Jan 17 at 9:21
  • I see. The problem is that Lua must return the lowercase for ÆØÅ. I'll see if Michals new modul will help or if I should handle these special characters individually. – user187802 Jan 17 at 11:23
5

I've just created a new package that adds support for the Unicode collation algorithm for LuaTeX - Lua-UCA. I've already added support for few languages, like Czech, German or Norwegian. We can use it instead of Xindex built in sorting mechanism.

Try the following version of xindex-norsk.lua:

-----------------------------------------------------------------------
--         FILE:  xindex-norsk.lua
--  DESCRIPTION:  configuration file for xindex.lua
-- REQUIREMENTS:  
--       AUTHOR:  Herbert Voß
--     MODIFIED:  Sveinung Heggen (2020-01-02)
--      LICENSE:  LPPL1.3
-----------------------------------------------------------------------

if not modules then modules = { } end modules ['xindex-cfg'] = {
      version = 0.20,
      comment = "configuration to xindex.lua",
       author = "Herbert Voss",
    copyright = "Herbert Voss",
      license = "LPPL 1.3"
}

local ducet = require "lua-uca.lua-uca-ducet"
local collator = require "lua-uca.lua-uca-collator"
local languages = require "lua-uca.lua-uca-languages"
local collator_obj = collator.new(ducet)

local language = "en" -- default language
-- language specified on the command line doesn't seem to be available
-- in the config file, so we just try to find it ourselves
for i, a in ipairs(arg) do
  if a == "-l" or a=="--language" then
    language = arg[i+1]
    break
  end
end

if languages[language] then
  print("[Lua-UCA] Loading language: " .. language)
  collator_obj = languages[language](collator_obj)
end

local upper = unicode.utf8.upper


escape_chars = { -- by default " is the escape char
  {'""', "\\escapedquote",      '\"{}' },
  {'"@', "\\escapedat",         "@"    },
  {'"|', "\\escapedvert",       "|"    },
  {'"!', "\\escapedexcl",       "!"    },
  {'"(', "\\escapedparenleft",  "("   },
  {'")', "\\escapedparenright", ")"  }
}

itemPageDelimiter = ","     -- Hello, 14
compressPages     = true    -- something like 12--15, instead of 12,13,14,15. the |( ... |) syntax is still valid
fCompress         = true    -- 3f -> page 3, 4 and 3ff -> page 3, 4, 5
minCompress       = 3       -- 14--17 or 
numericPage       = true    -- for non-numerical page numbers, like "VI-17"
sublabels         = {"", "---\\,", "--\\,", "-\\,"} -- for the (sub(sub(sub-items  first one is for item
pageNoPrefixDel   = ""     -- a delimiter for page numbers like "VI-17"
indexOpening      = ""     -- commands after \begin{theindex}
rangeSymbol       = "--"
idxnewletter      = "\\textbf"  -- Only valid if -n is not set

folium = { 
  de = {"f.", "ff."},
  en = {"f.", "ff."},
  fr = {"\\,sq","\\,sqq"},
  no = {"\\,f.","\\,ff."},
} 

function UTFCompare(a,b)
  local A = a["SortKey"]
  local B = b["SortKey"]
  return collator_obj:compare_strings(A,B)
end

function SORTendhook(list)
  -- get the headers for letter groups
  for k,v in ipairs(list) do 
    -- the collator:get_lowest_char will return character on the given
    -- position. It will be lowercase and without accents.
    local codepoints = collator_obj:string_to_codepoints(v.Entry)
    local codes = collator_obj:get_lowest_char(codepoints, 1)
    local sort_char = utf8.char(table.unpack(codes))
    v.sortChar = upper(sort_char) -- use unicode.utf8.upper to make the char uppercase
  end
  return list
end

--[[
    Each character's position in this array-like table determines its 'priority'.
    Several characters in the same slot have the same 'priority'.
]]

alphabet_lower = { --   for sorting
    { ' ' },  -- only for internal tests
    { 'a', 'á', 'à', },
    { 'b' },
    { 'c', 'ç' },
    { 'd' },
    { 'e', 'é', 'è', 'ë', 'ê' },
    { 'f' },
    { 'g' },
    { 'h' },
    { 'i', 'í', 'ì', 'î', 'ï' },
    { 'j' },
    { 'k' },
    { 'l' },
    { 'm' },
    { 'n', 'ñ' },
    { 'o', 'ó', 'ò', 'ô' },
    { 'p' },
    { 'q' },
    { 'r' },
    { 's', 'š', 'ß' },
    { 't' },
    { 'u', 'ú', 'ù', 'û' },
    { 'v' },
    { 'w' },
    { 'x' },
    { 'y', 'ý', 'ÿ', 'ü' },
    { 'z', 'ž' },
    { 'æ', 'œ', 'ä' },
    { 'ø', 'ö' },
    { 'å' }
}
alphabet_upper = { -- for sorting
    { ' ' },
    { 'A', 'Á', 'À', 'Â'},
    { 'B' },
    { 'C', 'Ç' },
    { 'D' },
    { 'E', 'È', 'É', 'Ë', 'Ê' },
    { 'F' },
    { 'G' },
    { 'H' },
    { 'I', 'Í', 'Ì', 'Ï', 'Î' },
    { 'J' },
    { 'K' },
    { 'L' },
    { 'M' },
    { 'N', 'Ñ' },
    { 'O', 'Ó', 'Ò', 'Ô' },
    { 'P' },
    { 'Q' },
    { 'R' },
    { 'S', 'Š' },
    { 'T' },
    { 'U', 'Ú', 'Ù', 'Û' },
    { 'V' },
    { 'W' },
    { 'X' },
    { 'Y', 'Ý', 'Ÿ', 'Ü' },
    { 'Z', 'Ž' },
    { 'Æ', 'Œ', 'Ä' },
    { 'Ø', 'Ö' },
    { 'Å' }
}

The relevant code is this:

local ducet = require "lua-uca.lua-uca-ducet"
local collator = require "lua-uca.lua-uca-collator"
local languages = require "lua-uca.lua-uca-languages"
local collator_obj = collator.new(ducet)
local language = "en" -- default language
-- language specified on the command line doesn't seem to be available
-- in the config file, so we just try to find it ourselves
for i, a in ipairs(arg) do
  if a == "-l" or a=="--language" then
    language = arg[i+1]
    break
  end
end

if languages[language] then
  print("[Lua-UCA] Loading language: " .. language)
  collator_obj = languages[language](collator_obj)
end

local upper = unicode.utf8.upper

function UTFCompare(a,b)
  local A = a["SortKey"]
  local B = b["SortKey"]
  return collator_obj:compare_strings(A,B)
end

function SORTendhook(list)
  -- get the headers for letter groups
  for k,v in ipairs(list) do 
    -- the collator:get_lowest_char will return character on the given
    -- position. It will be lowercase and without accents.
    local codepoints = collator_obj:string_to_codepoints(v.Entry)
    local codes = collator_obj:get_lowest_char(codepoints, 1)
    local sort_char = utf8.char(table.unpack(codes))
    v.sortChar = upper(sort_char) -- use unicode.utf8.upper to make the char uppercase
  end
  return list
end

It loads the needed libraries, creates the sorting object and applies the Norwegian rules. The UTFSort function is used by Xindex. We redefine it to use our sorting function. I've found that sorting works, but there is one problem - the first letters are not handled correctly, so Xindex produced separate headings for uppercase and lowercase letters. This is handled in the SORTendhook function.

This is the result:

enter image description here

|improve this answer|||||
  • Fantastic. I will test it later in the evening. – Sveinung Jan 13 at 12:33
  • @Sveinung sure, it is not yet part of TeX Live, it needs to be installed from github.com/michal-h21/lua-uca – michal.h21 Jan 13 at 14:21
  • @Sveinung does the installation fail? You may need to create the data directory manually and place file allkeys here. It is then necessary to execute command texlua tools/make_ducet.lua > src/lua-uca/lua-uca-ducet.lua. The files from src/lua-uca directory then need to be copied to the local TEXMF tree, which may need to be created manually. – michal.h21 Jan 13 at 14:50
  • There is no "make install" in MikTeX, so I have to try the manual way. – Sveinung Jan 13 at 16:52
  • @Sveinung oh, that's bad. I hope the method from my previous comment works. – michal.h21 Jan 13 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.