Use xindy to sort by bible book instead of ABC – or: How to add custom letter groups?

I try to generate an index which entries are not group by normal letters A, B, C, … but by books of the bible eg. Num, Dtn, Ps etc. So my question is how can I define a custom list of letter groups based on the books of the bible. As far as I understand the xindy manual all I have to do is to add some new letter groups with define-letter-groups in a custom style file, but my attempt failed.

This is my M(not)WE:

% !TeX program = xelatex

\begin{filecontents}{index-style.xdy}
(require "tex/inputenc/utf8.xdy")
(require "texindy.xdy")

(require "page-ranges.xdy")
(require "book-order.xdy")

(markup-locclass-list :open ": ")

(define-letter-groups ("Num"  "Dtn" "Ps"))
\end{filecontents}

\documentclass[a6paper,DIV=20]{scrartcl}

\usepackage[xindy]{imakeidx}

\makeindex[options = -M index-style -C utf8]

\begin{document}
Test
\index{Ps 1}
\index{Ps 10}
\index{Ps 2}
\index{Ps 1}
\index{Ps 3}
\index{Ps 56}
\index{Ps 5}
\index{Ps 34,1--2}
\index{Ps 34,7}
\index{Ps 34,1}
\index{Ps 34|full}
\index{Ps 34}
\index{Dtn 3,7.9}
\index{Dtn 3,5}
\index{Dtn 3,8}
\index{Num 122,3}
\index{Num 121}
\printindex
\end{document}


The xindy manual unfornutalty isn’t really details about letter groups and as far is I can tell I did nothing wrong in my style file, but I fear im missing something important.

What I found here on TeX.SX:

• xindy : how to add new letters and lettergroups to the alphabet? has an two answers but only for the special problem of adding groups for the Pali language. I’m not able to understand the first answer and transfer it to my problem. The second answer is a little less complicated but it doesn’t even work if I try to reproduce the results from there if I follow the steps one by one. I.e. the code – which is from 2011 – seems not to work.
• The answer for Xindy doesn't sort keywords properly suggest that I have to define a new language first. Is this true? Then why isn’t that mentioned in the xindy manual …?
• Index of bible verses with xindy has answers but no real solution …
• The answer on xindy: rule to remove index entry also uses a custom letter group but additionally with a strange (what kind of special sequence is ~b?) merge rule. However it seems to be a workaround if I use prefixes that aren’t normal letters:

(define-letter-groups ("Num" "Dtn" "Ps"))

(define-letter-group "Dtn" :prefixes ("#"))
(merge-rule "^Dtn" "#")

(define-letter-group "Num" :prefixes ("@"))
(merge-rule "^Num" "@")

(define-letter-group "Ps" :prefixes ("<"))
(merge-rule "^Ps" "<")


But this can’t be the intended solution …

• The answer on Strange sorting of Greek letters using glossaries with xindy uses new letter groups for greek letters and in that case no merge rule are used.

The last two questions led me to the theory that the problem with my letter groups is that they consist only of normal letters and are ignored because there are already groups for normal letters. However the xindy manual says

More problems arise when using languages with different letter schemes. Hungarian is an example. In Hungarian indexes the words beginning with the letters Cs, Ly, Ny and more are printed in a separate block. The words beginning with Ly, for example, appear behind the words beginning with an L. xindy allows to define this kind of letter groups as well. Add the following lines to the style file.

     (define-letter-group "ly" :after "l" :before "m")
(define-letter-group "ny" :after "n" :before "o")

:capitalize)


Which is obviously an example with only letters …

Another workaround is to only use merge rules and map each book to a letter of the alphabet and then add some TeX code to transform \lettergroup{A} to output Num.

Edit: Please consider the bible books only as one example for the usage of new letter groups. I’m not looking for further workarounds to typeset/index the bible books, but a general solution on how to define letter groups with xindy.

• Imho xindy has some rather serious flaws -- including bad documentation, no real utf8 support and bad integration in the tlmgr structure. The curious prefixes used for the sorting are one indication of the unsolved problems. Do you really need to use it? Can't you use glossaries? – Ulrike Fischer Mar 5 '17 at 14:25
• @UlrikeFischer: I thought it was developed for use with native Unicode support and had no problems with this. The other flaws are there – unfortunately. I’m not bound to xindy but i found it’s style system a bit more intuitive and flexible than the one of makeindex. and glossaries uses makeindex or xindy in the background doesn’t it? I never used it before … – Tobi Mar 5 '17 at 19:47
• "No real utf8" was a bit harsh, but xindy was developed at a time when most pdflatex users still used 8-bit encoding and so e.g. the utf8.xdy are 8-bit encoded and not utf8. Nicola is currently trying to develop a tool for glossaries, I saw the announcement. – Ulrike Fischer Mar 5 '17 at 21:26
• @UlrikeFischer: Hm … I give this question another try on the xindy mailing list. Does anybody know if the development is still active? – Tobi Mar 5 '17 at 21:59

The easiest solution is to use Klingon: change the line

\makeindex[options = -M index-style -C utf8]


\makeindex[options = -M index-style -C utf8 -L klingon]


This produces a PDF file whose index page is typeset as:

which I believe is what you wanted.

This one was, for me, a good lesson in software testing: I had to treat the xindy program almost as a black-box, because its documentation is terrible and even actively misleading. In fact, of all the time I spent trying to figure this out, the biggest leap towards a solution was the moment I decided to completely stop trusting the documentation.

The following is not exactly how I arrived at the solution/understanding, but a lightly “fictionalized account” of how one might have.

To recap the question: After saving the LaTeX source from the question as question.tex, changing filecontents to filecontents* (so that it doesn't write %% comments to the file), and running xelatex -shell-escape question.tex (note: shell-escape is in general dangerous; you should not use that option without being absolutely sure the file is safe to run), we get a typeset PDF whose index page looks like the following:

So the problem is that it seems to have completely ignored the (define-letter-groups ... in the index style.

If you look in the log file (question.log), it mentions the program that was invoked:

runsystem(texindy -M index-style -C utf8 question.idx)...executed.


And you can verify for yourself that it is this command (texindy -M index-style -C utf8 question.idx) that turns question.idx:

\indexentry{Ps 1}{1}
\indexentry{Ps 10}{1}
\indexentry{Ps 2}{1}
\indexentry{Ps 1}{1}
\indexentry{Ps 3}{1}
\indexentry{Ps 56}{1}
\indexentry{Ps 5}{1}
\indexentry{Ps 34,1--2}{1}
\indexentry{Ps 34,7}{1}
\indexentry{Ps 34,1}{1}
\indexentry{Ps 34|full}{1}
\indexentry{Ps 34}{1}
\indexentry{Dtn 3,7.9}{1}
\indexentry{Dtn 3,5}{1}
\indexentry{Dtn 3,8}{1}
\indexentry{Num 122,3}{1}
\indexentry{Num 121}{1}


into question.ind:

\begin{theindex}
\providecommand*\lettergroupDefault[1]{}
\providecommand*\lettergroup[1]{%
\par\textbf{#1}\par
\nopagebreak
}

\lettergroup{D}
\item Dtn 3,5: 1
\item Dtn 3,7.9: 1
\item Dtn 3,8: 1

\indexspace

\lettergroup{N}
\item Num 121: 1
\item Num 122,3: 1

\indexspace

\lettergroup{P}
\item Ps 1: 1
\item Ps 2: 1
\item Ps 3: 1
\item Ps 5: 1
\item Ps 10: 1
\item Ps 34: 1
\item Ps 34,1: 1
\item Ps 34,1--2: 1
\item Ps 34,7: 1
\item Ps 56: 1

\end{theindex}


The bug already manifests here. So at this point we can forget about the TeX stuff and focus on getting the desired question.ind out of the question.idx using the texindy (or another) command.

The texindy command has some debug options. For example, adding --debug script shows that texindy is essentially calling xindy as xindy -d script -L general -C utf8 -M tex/inputenc/utf8 -M texindy -M page-ranges -M word-order -M index-style.xdy -I latex question.idx. We can run that directly, and see that it produces the same output (the “bad” question.ind).

This list of modules automatically included by texindy also suggests that some of the lines of our index-style.xdy are redundant: we can boil it down to just

(markup-locclass-list :open ": ")
(define-letter-groups ("Num"  "Dtn" "Ps"))


and keep the output the same. We can go further and also reduce the xindy command to xindy -M texindy -M index-style question.idx. Now we can run xindy directly, instead of texindy.

To continue exploring the debug options, the first thing is to have xindy log to a file instead of throwing it away (to /dev/null). This is done with -t question.ilg. This log file has output like

Forming letter-groups:
Letter-group: "?? 0000001" -> "P"
Letter-group: "?? 0000010" -> "P"


and so on, which clearly correspond one-to-one with the lines from the input (question.idx). (?? is what it displays as in my terminal, but I can open it up in a editor to see the actual bytes: e.g. less shows the second line as Letter-group: "<C8><D0> 0000010" -> "P" with those two characters higlighted.)

So, for some reason, it has converted input entry like \indexentry{Ps 10}{1} into "<C8><D0> 0000010", which then gets mapped to letter group P.

The --debug level=1 (or higher) option to xindy has something relevant in the log: lines like

Add sort rule to run 0: #<ordrule: '<E4>' => '<80>' :again NIL>
Add sort rule to run 0: #<ordrule: 'A' => '<80>' :again NIL>
...
Add sort rule to run 0: #<ordrule: 'p' => '<C8>' :again NIL>
Add sort rule to run 0: #<ordrule: 'P' => '<C8>' :again NIL>


and so on. But this still doesn't explain where those sort rules are coming from. With the remaining debug options --debug script --debug keep_tmpfiles though, we can find out: First, the command uses a filter called tex2xindy to convert our .idx file into a file that has lines like:

(indexentry :tkey (("Ps 1")) :locref "1")
(indexentry :tkey (("Ps 10")) :locref "1")


(entirely in Lisp syntax). This file, we will soon learn, is called the "rawindex". Then, the “core” of xindy runs with a command like:

xindy.run -M /Library/TeX/texbin/xindy.mem -E iso-8859-1 -x (progn
(searchpath ".:/usr/local/texlive/2015/texmf-dist/xindy/modules:/usr/local/texlive/2015/texmf-dist/xindy/modules/base")
(xindy:startup
:idxstyle "<tmp file 1>"
:rawindex "<above tmp file>"
:output "./question.ind"
:logfile "question.ilg"

)
(exit))


We can run this command directly, and see that it (still) produces the same .ind file. We're getting close: looking at the tmp file above (the attribute :idxstyle), it looks like:

(require "lang/general/latin9-lang.xdy")
(require "texindy.xdy")
(require "index-style.xdy")


The last two (of the three) lines were specified manually by our commandline xindy -M texindy -M index-style; the first one was inserted automatically by xindy. (The latin9 does not matter here; if we had used -C utf8 we'd get lang/general/utf8-lang.xdy here in which the problems are the same.) Looking inside that file (texmf-dist/xindy/modules/lang/general/latin9-lang.xdy), it starts with

(require "lang/general/latin9.xdy")


and that file starts with lines like:

(define-letter-group "A" :prefixes ("<80>"))
(define-letter-group "B" :after "A" :prefixes ("<84>"))
(define-letter-group "C" :after "B" :prefixes ("<86>"))
(define-letter-group "D" :after "C" :prefixes ("<8D>"))


which give a hint as to what's going on: already by the time define-letter-group is involved, the characters have already been transformed into bytes like "<80>" (recall the log lines like Add sort rule to run 0: #<ordrule: 'A' => '<80>' :again NIL> that we saw earlier). So these define-letter-group commands are operating not on the raw input in the index file, but on these transformed bytes.

At this point, we can just use the same byte replacements as in this file, to get Solution 1: Create the file index-style.xdy whose contents are lightly modified from your original, using the following Python script:

s = '''
(markup-locclass-list :open ": ")

(define-letter-group "Num" :prefixes ("\xbc\xe0\xbb"))
(define-letter-group "Dtn" :prefixes ("\x8d\xda\xbc") :after "Num")
(define-letter-group "Ps" :prefixes ("\xc8\xd0") :after "Dtn")
'''

open('index-style.xdy', 'w').write(s)


(This Python script just assigns the file contents as a string to the variable s and writes out that variable to the file index-style.xdy: you can use any language or even a shell script; the goal is just to get binary data into the file.)

Then using this index-style.xdy gives the index you desired (and which is in the screenshot above).

The explanation, for why these transformations happen, is lower in the same file (now I'm using utf8.xdy instead of latin9.xdy):

(define-rule-set "xy-alphabetize"

:rules  (("À" "<80>" :string)
("Ă" "<80>" :string)
("â" "<80>" :string)
...
("A" "<80>" :string)
...
))


and this rule-set is used in utf8-lang.xdy:

(use-rule-set :run 0
:rule-set ("xy-alphabetize" "xy-ignore-special"))


because of which in the very first (0th) run, all the characters are transformed already, which is why our (define-letter-groups ("Num" "Dtn" "Ps")) did not work. The documentation is correct about (define-letter-groups ...), in the sense that the feature really exists and the core of xindy supports it. However, the default “general” configuration (which by the way, is documented in the file comments as “A general sorting order for Western European languages”), explicitly does some stuff that gets in the way, and the documentation says nothing about dealing with this.

If they had provided instead of “general” a “bare” “language” that is even more minimal than “general” (and does not do the hackery with translating characters to bytes outside the ascii range), then using (define-letter-groups with english-alphabet characters would have just worked (as simple as giving -L bare or whatever).

To confirm that this is the case, we can pursue an alternative approach (Solution 2), of giving an explicit file (without the first line), instead of the temporary file for idxstyle. In this file we remove the first (require "lang/general/latin9-lang.xdy") line and keep only the requires of texindy and our index-style.xdy. If we do that, e.g. I put the two lines into a file called forced-idxstyle and changed the xindy.run command to just

/Library/TeX/texbin/xindy.run -M /Library/TeX/texbin/xindy.mem -E iso-8859-1 -x '(progn (searchpath ".:/usr/local/texlive/2015/texmf-dist/xindy/modules:/usr/local/texlive/2015/texmf-dist/xindy/modules/base") (xindy:startup :idxstyle "./forced-idxstyle" :rawindex "<same tmp file as before>" :output "./question.ind" :logfile "question.ilg") (exit))'


it worked: our plain index-style.xdy file which had just the two lines

(markup-locclass-list :open ": ")
(define-letter-groups ("Num"  "Dtn" "Ps"))


now starts giving the exact question.ind file we want (and we can run xelatex and we'll get the desired typeset output in the index).

This confirms that define-letter-groups works as documented, and would have been usable by the end user (us) if not for all the “helpful” transformations done by all of xindy's default languages, for treating accented letters the same as unaccented letters.

Actually not all: among the languages that come with xindy there are a few that don't work with Latin-script characters, and in particular don't transform "A" (for example) into anything else:

belarusian bulgarian georgian greek hebrew klingon korean macedonian russian serbian ukrainian

Of these, if you really want a language that will never get in the way when using English, then “korean” is the most promising, because the others (even Klingon which isn't even in Unicode and uses private-area characters!) do transform some “special” characters like hyphens and semicolons. So this gives Solution 3 that I put at the top of this answer: keep your original index style, keep your original TeX file, just append -L korean to the indexing command, and you get the index you desired. (The other languages like Klingon etc. also work.)

• Again: Thank you very much for your research on this topic. Although the “klingon solution” is more or less a workaround and i didn’t wanted to use workarounds, this is the way I go. There’s only one problem: Books like 1 Kor will be sorted in the default group since even the klingons seem to know how to sort numbers. Therefor I use I Kor as group and a merge-rule for the 1 Kor entries. – Tobi Mar 16 '17 at 1:23

I solved this problem for my work without using xindy. My alternative was to use sort keys. Thus: \index{19@Psalms!0901 @9:1}. 19 is Psalms' order in the Hebrew Bible, so this number before the @ sign sorts the books in correct order. The exclamation point (!) makes the book itself, Psalms, the item and verses will be indexed as subitems below it. You can just remove this if you want the book title before every citation. 0901 is the sort code for 9:1 with zeroes preventing chapters from 10 (example: 11:1) being listed first. The space prevents a verse range from appearing before single verses, so 0901_ will sort ahead of 0901-07. (This is a genius insight of Barbara Beeton's from another post).

If you have \index{05@Deuteronomy!1212 @12:12} \index{02@Exodus!0101-12@1:1-12} \index{19@Psalms!0901 @9:1} this should now result in correct biblical order. So:

Exodus

1:1-12

Deuteronomy

12:12

Psalms

9:1

I preferred this solution myself. You may also want to check out the bibleref pacakage too. I haven't used it, but I believe it can work with indexes.

• Thanks for your answer but I need a solution without manual sort keys (there are lots of index entries, which can’t be redone manually …). bibleref would also nee too much manual changes (the book is finished) and furthermore doesn’t seem to support things like Ps 90(91),1.4. – Tobi Mar 1 '17 at 10:42
• Fair enough. My book was also already done. Do I understand correctly that all the citations are already indexed in the form cited in your example? If you can't use xindy in the end, you can use find and replace to change the indexing code: \index{Dtn 3,8} can be changed to \index{04@Deuteronomy 3,8} without too much trouble, as backwards as such a fix may be. Hopefully it won't come to something like that! Looking forward to seeing other answers to this problem. – niesen Mar 1 '17 at 11:04
• Sure I can do a lot by searcher an replace even dynamically. But I want to know why the new letter groups don’t work and my example is just one application for this … – Tobi Mar 1 '17 at 12:49