Indexing through texindy in imakeidx with custom .xdy style fails to produce desired order in index

Question

I am trying to produce an index using texindy through the imakeidx make that follows the Sanskrit alphabetical order. I am working in XeLaTeX.

I created a custom .xdy style file to be called as an option to texindy, which sets up letter groups for all of the (sequences of) Roman letters (many of which have diacritics), and defined the ordering of all letter groups with respect to one another, following the example at http://www.xindy.org/doc/tutorial-3.html (subsection 3.3).

The index that is produced, however, simply seems to follow a standard Latin alphabetical order (though it indexes characters with diacritics before characters without diacritics. For instance, because "dh" is a distinct letter from "d", which follows "d" in Sanskrit, all words beginning with "dh" should follow all words beginning with "d" + any other letter. For example "drapsa-" should precede "dhanvan-" (NB that "h" is also a letter, but it is the final letter in the alphabet, so "dhanvan-" would follow "drapsa-" in any case, even in "dh" were not defined as a letter group). Please ask me if you need more detail about my expectation for the index, though I think that the text of my .xdy below should make my intentions reasonably clear.

I give below here: 1) my .tex code with some example words to be indexed; 2) my .xdy file; 3) the text of my log relevant to the index as the .tex compiles. I suspect that the problem may lie with some of the automatic modules that are called for processing the index, but I don't know how to disengage them without also disengaging some components that are needed.

1)

\documentclass[titlepage,letterpaper,twoside,12pt]{book}
\usepackage{fontspec}
\usepackage{xunicode}
\usepackage{xltxtra}
\usepackage{fontenc}[utf8x]
\setmainfont{Brill}
\usepackage{imakeidx}
\makeindex[program=texindy, options=-M skt2]
\newcommand\IndexR[1]{#1\index{#1}}
\begin{document}


\IndexR{amhas-} \IndexR{ám̐has-}, \IndexR{áti}, \IndexR{āsúr-}, \IndexR{íṣṭa-},
\IndexR{krátu-}, \IndexR{devī́-}, \IndexR{devī́nām}, \IndexR{dhánvan-},
\IndexR{drapsá-}, \IndexR{áśva-}, \IndexR{ohas-}, \IndexR{itá-}, \IndexR{īhate}

\printindex
\end{document}

2)

(require "tex/inputenc/utf8.xdy")
;; Here follow all letter-groups.

(define-letter-groups
("ṃ" "m̐" " "ḥ" "a" "ā" "i" "ī" "u" "ū" "e" "o" "ai" "au" "r̥" "r̥̄" "k" "kh"
 "g" "gh" "ṅ" "c" "ch" "j" "jh" "n͂" "t" "th" "d" "dh" "n" "ṭ" "ṭh" "ḍ" "ḍh"
 "ṇ" "p" "ph" "b" "bh" "m" "y" "r" "l" "v" "ś" "ṣ" "s" "h"))

(define-letter-group "ṃ" :before "m̐")
(define-letter-group "m̐" :after "ṃ" :before "ḥ")
(define-letter-group "ḥ" :after "m̐" :before "a")
(define-letter-group "a" :after "ḥ" :before "ā")
(define-letter-group "ā" :after "a" :before "i")
(define-letter-group "i" :after "ā" :before "ī")
(define-letter-group "ī" :after "i" :before "u")
(define-letter-group "u" :after "ī" :before "ū")
(define-letter-group "ū" :after "u" :before "e")
(define-letter-group "e" :after "ū" :before "o")
(define-letter-group "o" :after "e" :before "ai")
(define-letter-group "ai" :after "o" :before "au")
(define-letter-group "au" :after "ai" :before "r̥")
(define-letter-group "r̥" :after "au" :before "r̥̄")
(define-letter-group "r̥̄" :after "r̥" :before "k")
(define-letter-group "k" :after "r̥̄" :before "kh")
(define-letter-group "kh" :after "k" :before "g")
(define-letter-group "g" :after "kh" :before "gh")
(define-letter-group "gh" :after "g" :before "c")
(define-letter-group "ṅ" :after "gh" :before "c")
(define-letter-group "c" :after "ṅ" :before "ch")
(define-letter-group "ch" :after "c" :before "j")
(define-letter-group "j" :after "ch" :before "jh")
(define-letter-group "jh" :after "j" :before "n͂")
(define-letter-group "n͂" :after "jh" :before "t")
(define-letter-group "t" :after "n͂" :before "th")
(define-letter-group "th" :after "t" :before "d")
(define-letter-group "d" :after "th" :before "dh")
(define-letter-group "dh" :after "d" :before "n")
(define-letter-group "n" :after "dh" :before "ṭ")
(define-letter-group "ṭ" :after "n" :before "ṭh")
(define-letter-group "ṭh" :after "ṭ" :before "ḍ")
(define-letter-group "ḍ" :after "ṭh" :before "ḍh")
(define-letter-group "ḍh" :after "ḍ" :before "ṇ")
(define-letter-group "ṇ" :after "ḍh" :before "p")
(define-letter-group "p" :after "ṇ" :before "ph")
(define-letter-group "ph" :after "p" :before "b")
(define-letter-group "b" :after "ph" :before "bh")
(define-letter-group "bh" :after "b" :before "m")
(define-letter-group "m" :after "bh" :before "y")
(define-letter-group "y" :after "m" :before "r")
(define-letter-group "r" :after "y" :before "l")
(define-letter-group "l" :after "r" :before "v")
(define-letter-group "v" :after "l" :before "ś")
(define-letter-group "ś" :after "v" :before "ṣ")
(define-letter-group "ṣ" :after "ś" :before "s")
(define-letter-group "s" :after "ṣ" :before "h")
(define-letter-group "h" :after "s")

(markup-letter-group :open-head "~n {\bf " :close-head "}")


;;
;; The sort-rules map all letters to their lowercase counterpart.
;;

(sort-rule "A" "a")
(sort-rule "B" "b")
(sort-rule "C" "c")
(sort-rule "D" "d")
(sort-rule "E" "e")
(sort-rule "F" "f") 
(sort-rule "G" "g")
(sort-rule "H" "h")
(sort-rule "I" "i")
(sort-rule "J" "j")
(sort-rule "K" "k")
(sort-rule "L" "l")
(sort-rule "M" "m")
(sort-rule "N" "n")
(sort-rule "O" "o")
(sort-rule "P" "p")
(sort-rule "Q" "q")
(sort-rule "R" "r")
(sort-rule "S" "s")
(sort-rule "T" "t")
(sort-rule "U" "u")
(sort-rule "V" "v")
(sort-rule "W" "w")
(sort-rule "X" "x")
(sort-rule "Y" "y")
(sort-rule "Z" "z")
;; Accented Sorting
(sort-rule "á" "a")
(sort-rule "ā́" "ā")
(sort-rule "í" "i")
(sort-rule "ī́" "ī")
(sort-rule "ú" "u")
(sort-rule "ū" "ū")
(sort-rule "é" "e")
(sort-rule "ó" "o")
(sort-rule "ái" "ai")
(sort-rule "áu" "au")
(sort-rule "ŕ̥" "r̥")
(sort-rule "r̥̄́" "r̥̄")

3)

Reading indexstyle...
Loading module "/var/folders/1t/1by7npx95zdfkspc6fp0w8nm0000gn/T/jN96RZb09V"...
Loading module "lang/general/latin9-lang.xdy"...
Loading module "lang/general/latin9.xdy"...
Finished loading module "lang/general/latin9.xdy".
Finished loading module "lang/general/latin9-lang.xdy".
Loading module "tex/inputenc/latin.xdy"...
Finished loading module "tex/inputenc/latin.xdy".
Loading module "texindy.xdy"...
Loading module "numeric-sort.xdy"...
Finished loading module "numeric-sort.xdy".
Loading module "latex.xdy"...
Loading module "tex.xdy"...
Finished loading module "tex.xdy".
Finished loading module "latex.xdy".
Loading module "latex-loc-fmts.xdy"...
Finished loading module "latex-loc-fmts.xdy".
Loading module "makeindex.xdy"...
Finished loading module "makeindex.xdy".
Finished loading module "texindy.xdy".
Loading module "page-ranges.xdy"...
Finished loading module "page-ranges.xdy".
Loading module "word-order.xdy"...
Finished loading module "word-order.xdy".
Loading module "skt2.xdy"...
Finished loading module "skt2.xdy".
Finished loading module "/var/folders/1t/1by7npx95zdfkspc6fp0w8nm0000gn/T/jN96RZb09V".
Finished reading indexstyle.
Finalizing indexstyle... (done)

Reading raw-index "/var/folders/1t/1by7npx95zdfkspc6fp0w8nm0000gn/T/SQs8on3FCq"...
Finished reading raw-index.

Processing index... [10%] [20%] [30%] [40%] [50%] [60%] [70%] [80%] [90%] [100%]
Finished processing index.

Writing markup... [10%] [20%] [30%] [40%] [50%] [60%] [70%] [80%] [90%] [100%]
Markup written into file "./IndexTest2.ind".
(./IndexTest2.ind) [3] (./IndexTest2.aux) )
Output written on IndexTest2.pdf (3 pages).
Transcript written on IndexTest2.log.

Answer 1

Experimental; first version (IAST scheme)

I am trying to setup this transliteration in Xindy. I've used lowercase characters for letter groups, but it isn't typical in European languages. I've used merge-rules for capital letters, we don't use these rules this way either. So we only see Shiva not shiva after this step.

I disliked rendering of several double accented letters in Latin Modern, therefore I use Code2000 font. We can run a list of alphabet letters and an example in xelatex and lualatex. I was using Wikipeadia to verify the order of letters with respect to Sanskrit Keyboard Online, I used several more letter combinations, but it looked I was doing more harm than good, so I withdrew them.

First, I enclose the alphabet:

% run: {xe|lua}latex typesetme.tex
\documentclass[a4paper]{article}
\pagestyle{empty}
\usepackage{fontspec}
\setmainfont{Code2000}
\newenvironment{alphabet}{\begin{tabular}{*{16}{l}}}{\end{tabular}}
\addtolength{\voffset}{-0.5in}
\addtolength{\textheight}{1in}

\begin{document}
\section{Sanskrit (transliteration)}
\subsection{Alphabet}
\begin{alphabet}
a\,A & á\,Á\\
ā\,Ā & ā́\,Ā́\\
i\,I & í\,Í\\
ī\,Ī & ī́\,Ī́\\
u\,U & ú\,Ú\\
ū\,Ū\\
ṛ\,Ṛ & r̥\,R̥ & ŕ̥\,Ŕ̥\\
ṝ\,Ṝ & r̥̄\,R̥̄ & r̥̄́\,R̥̄́\\
ḷ\,Ḷ\\
ḹ\,Ḹ\\
e\,E & é\,É\\
ai\,AI & ái\,ÁI\\
o\,O & ó\,Ó\\
au\,AU & áu\,ÁU\\
ṃ\,Ṃ\\
m̐\,M̐\\
ḥ\,Ḥ\\
k\,K\\
kh\,KH\\
g\,G\\
gh\,GH\\
ṅ\,Ṅ\\
c\,C\\
ch\,CH\\
j\,J\\
jh\,JH\\
ñ\,Ñ\\
ṭ\,Ṭ\\
ṭh\,ṬH\\
ḍ\,Ḍ\\
ḍh\,ḌH\\
ṇ\,Ṇ\\
t\,T\\
th\,TH\\
d\,D\\
dh\,DH\\
n\,N\\
p\,P\\
ph\,PH\\
b\,B\\
bh\,BH\\
m\,M\\
y\,Y\\
r\,R\\
l\,L\\
v\,V\\
ś\,Ś\\
ṣ\,Ṣ\\
s\,S\\
h\,H
\end{alphabet}

\ifx\relax
\subsection{Ligatures}
\begin{flushleft}
None.
\end{flushleft}
\subsubsection{Upper-/lowercase words}
Capitalized or uppercase words are sorted \emph{before} otherwise equal lowercase words.
\subsubsection{Special characters}
The order of special characters and letters is:
\begin{flushleft}
?\hspace{4mm}!\hspace{4mm}.\hspace{4mm}letters\hspace{4mm}-\hspace{4mm}'
\end{flushleft}
\fi
\end{document}

Usage

Please download these two files to your working directory:

wget http://striz7.fame.utb.cz/tex-sx/sa/version1/utf8.pl-sa.in.xdy
wget http://striz7.fame.utb.cz/tex-sx/sa/version1/utf8.pl-sa.in-test.xdy

In case you would like to compare the vector versions, please download these four files too:

wget http://striz7.fame.utb.cz/tex-sx/sa/version1/typesetme.pdf
wget http://striz7.fame.utb.cz/tex-sx/sa/version1/typesetme.tex
wget http://striz7.fame.utb.cz/tex-sx/sa/version1/mal-sanskrit.pdf
wget http://striz7.fame.utb.cz/tex-sx/sa/version1/mal-sanskrit.tex

I enclose an example and a preview of the PDF file which I've got after running these three commands:

lualatex mal-sanskrit.tex
xindy -M texindy -M utf8.pl-sa.in-test -M bonus mal-sanskrit.idx
lualatex mal-sanskrit.tex

% run: xelatex or lualatex
%lualatex mal-sanskrit.tex
%xindy -M texindy -M utf8.pl-sa.in-test -M bonus mal-sanskrit.idx
%lualatex mal-sanskrit.tex
\documentclass[a4paper]{article}
\pagestyle{empty}
%\usepackage{luatextra} % for lualatex
%\usepackage{xltxtra} % xelatex
\usepackage{fontspec}
\newfontfamily{\codetwo}{Code2000}% Brill
\usepackage[noautomatic]{imakeidx}
\indexsetup{firstpagestyle=empty}
\usepackage[colorlinks]{hyperref}
\makeindex
\newcommand\IndexR[1]{{\codetwo#1}\index{#1}}
\usepackage{filecontents}
\def\malformat#1{{\Large#1}\smallskip}

\begin{filecontents*}{bonus.xdy}
(markup-letter-group :open-head "~n  \malformat{" :close-head "}")
(merge-rule "A" "a")
(merge-rule "Á" "á")
(merge-rule "Ā" "ā")
(merge-rule "Ā́" "ā́")
(merge-rule "I" "i")
(merge-rule "Í" "í") 
(merge-rule "Ī" "ī")  
(merge-rule "Ī́" "ī́") 
(merge-rule "U" "u")  
(merge-rule "Ú" "ú") 
(merge-rule "Ū" "ū") 
(merge-rule "Ṛ" "ṛ")
(merge-rule "R̥" "r̥")  
(merge-rule "Ŕ̥" "ŕ̥") 
(merge-rule "Ṝ" "ṝ")  
(merge-rule "R̥̄" "r̥̄")  
(merge-rule "R̥̄́" "r̥̄́") 
(merge-rule "Ḷ" "ḷ") 
(merge-rule "Ḹ" "ḹ") 
(merge-rule "E" "e")  
(merge-rule "É" "é") 
(merge-rule "AI" "ai")  
(merge-rule "ÁI" "ái") 
(merge-rule "O" "o")  
(merge-rule "Ó" "ó") 
(merge-rule "AU" "au")  
(merge-rule "ÁU" "áu") 
(merge-rule "Ṃ" "ṃ") 
(merge-rule "M̐" "m̐") 
(merge-rule "Ḥ" "ḥ") 
(merge-rule "K" "k") 
(merge-rule "KH" "kh") 
(merge-rule "G" "g") 
(merge-rule "GH" "gh") 
(merge-rule "Ṅ" "ṅ") 
(merge-rule "C" "c") 
(merge-rule "CH" "ch") 
(merge-rule "J" "j") 
(merge-rule "JH" "jh") 
(merge-rule "Ñ" "ñ") 
(merge-rule "Ṭ" "ṭ") 
(merge-rule "ṬH" "ṭh") 
(merge-rule "Ḍ" "ḍ") 
(merge-rule "ḌH" "ḍh") 
(merge-rule "Ṇ" "ṇ") 
(merge-rule "T" "t") 
(merge-rule "TH" "th") 
(merge-rule "D" "d") 
(merge-rule "DH" "dh") 
(merge-rule "N" "n") 
(merge-rule "P" "p") 
(merge-rule "PH" "ph") 
(merge-rule "B" "b") 
(merge-rule "BH" "bh") 
(merge-rule "M" "m") 
(merge-rule "Y" "y") 
(merge-rule "R" "r") 
(merge-rule "L" "l") 
(merge-rule "V" "v") 
(merge-rule "ś" "Ś") 
(merge-rule "Ṣ" "ṣ") 
(merge-rule "S" "s") 
(merge-rule "H" "h") 
\end{filecontents*}


\begin{document}
Text before.
% From TeX.SX...
\IndexR{amhas-} \IndexR{ám̐has-}, \IndexR{áti}, \IndexR{āsúr-}, \IndexR{íṣṭa-},
\IndexR{krátu-}, \IndexR{devī́-}, \IndexR{devī́nām}, \IndexR{dhánvan-},
\IndexR{drapsá-}, \IndexR{áśva-}, \IndexR{ohas-}, \IndexR{itá-}, \IndexR{īhate}
% From Wikipedia...
\newpage
\IndexR{ārṣa}, \IndexR{saṃskṛtám}, \IndexR{paścimottarī},  
\IndexR{madhyadeśī},  \IndexR{pūrvi}, \IndexR{dakṣiṇī}, 
\IndexR{prākṛta-}, \IndexR{Pāṇini}, \IndexR{Pāṇinian}, 
\IndexR{Janani}, \IndexR{Janmabhūmisca}, \IndexR{Svargādapi}, 
\IndexR{garīyasi}, \IndexR{Aṣṭādhyāyī}, \IndexR{vyākaraṇa}, 
\IndexR{Aṣṭādhyāyī}, \IndexR{Pāṇini}, \IndexR{Patañjali}, 
\IndexR{Mahābhāṣya},\IndexR{Aṣṭādhyāyī},\IndexR{Trimuni},
\IndexR{Vyākarana},\IndexR{Vārtikas},\IndexR{Jayaditya}, 
\IndexR{Vāmana}, \IndexR{Shiva},\IndexR{shiva}
\IndexR{Kāsikā}, \IndexR{Pāṇinian}, \IndexR{pāṇinian}, 
% Numbers
\newpage
\IndexR{éka-}, \IndexR{dva-}, \IndexR{tri-}, \IndexR{catúr-}, 
\IndexR{páñcan-}, \IndexR{ṣáṣ-}, \IndexR{saptán-}, \IndexR{aṣṭá-}, \IndexR{návan-}, \IndexR{dáśan-}
Text after.

\begingroup
\codetwo
\printindex
\endgroup
\end{document}

Answer 2

I would add "-I omega". Omega is a TeX system working internally in Unicode. XeLaTeX does exactly the same. Setting input markup to omega has the effect that UTF-8 is assumed and unnecessary modules for handling input are disabled. Some time ago I planned to make support for Indic languages. I only know Hindi and without help of others I cannot continue my work. My support for Hindi is here: http://icebearsoft.euweb.cz/xindy-devanagari/ I need a few more test files and someone who will verify the result.