(This is not a real question, it's rather a list of thoughts and ideas.)
I'm not a serious TeXist, but this topic of index creation interests me a lot. As I like programming in Lua, I started to think if I could try an experiment or two to cover some topics in the indexing world. Let me think about it for a while. In this post/question, I'm trying to collect ideas which we are missing, could be improved or simply it would be nice to have them among out-of-the-box features.
This list might help programmers to actually program something new or me to get on the right track before I actually do some programming.
What's ready for use? (TeX Live 2014, possibly next years too)
There are some tools and packages we use frequently for sorting and/or index creation:
- The MakeIndex index processor. There is a fine chapter in the LaTeX Companion, 2nd ed. about modifying styles.
- The Xindy index processor.
- The imakeidx package and some other packages, e.g. splitindex, multind.
- The glossaries package.
- The OPmac extended plainTeX macros (see
\makeindex). - The biber tool (sorting only).
- The ConTeXt engine.
What we may like to have at our disposal? (to be updated)
Support for more languages
For xindy, I tried to prepare definition files for Sanskrt (transliteration scheme) and Pāḷi. But it's quite time consuming to set it up properly. It would be better to define just some letters, let's say in a Lua table. It would be also helpful to have an option to set up a language dialect.
Support for CJKV languages
The support for CJKV and other exotic languages is weak in xindy. They are using radicals, number of strokes and even frequency of kanjis in Japanese (JLPT levels). It means that we must handle additional data sets, e.g. Unicode Character Database; UCD or Unihan Database. I haven't checked out the new engines and tools in detail (xepersian, support for Japanese, Chinese, Korean etc.), I cannot say if the index creation is ready for TeX users.
Setting up a new set of sorting rules and letter groups
One of the examples comes from the chemistry field and its Periodic Table of Elements. I believe we could find some examples in Astrophysics. It would be interesting to sort items by its colour (name, wave), but that's not a common task.
Two-way hyperlinks
It would be great to have two-way hyperlinks coming to/from the index. I tried that once in this question. Sometimes we turn two-way hyperlinks on elsewhere, e.g. in the footnotes.
Mixing writing
It might be possible to switch between transliteration schemes (like in XeLaTeX via map files) at the input/output side, or we could even mix them, e.g. sayōnara + さようなら which is the same index entry (romanized version, hiragana).
Index with different piece of information
I've shown one of the approaches in this question, but it is far from user-friendly. Instead of page numbers, we might have other blocks, such as sections, numbers of line in the source code, numbers of the problem/solution, numbers of some examples, verse numbers, or own counter. There is an interesting question dealing with adding a beginning of the verse to the index.
List of occurences
Also a list of occurences of the index entries (and number of index entries in the group letter) should be easily available (see this question).
Support for indexing more languages in one index
There is an open issue of finderlist, it might be useful not only for Pohnpeian + English. We should be able to change the order of languages of such a finderlist. There is one more interesting open issue: Ancient Egyptian, where could be even three different languages/sorting systems (pronunciation, transliteration, Unicode notation and translation).
Modifying rules
I tried to correct Icelandic and Vietnamese in Xindy, but it took a lot of time. In Czech, for instance, some people like to have letter groups as A, B, C, Č, D, ..., S, Š, ... U, V, W, X, Y, Z, but some people (due to lack of index entries) like to have A, B, CČ, D, ..., SŠ, ..., U, V, W, XYZ.
Testing files
In addition to the previous item, it would be great to have some testing files per language. Or some examples of / links to already created documents with index. It's possible to find some books, but it requires a lot of researching for non-native speakers in that language. It remains me test cases in data analysis to measure efficiency of a new algorithm.
Sorting engine on, indexing engine off
It would be nice to be able to switch between indexing and sorting engines. I tried to modify a xindy style once. Indexing processor might behave like a sorting engine upon request.
Sorting engine off, indexing engine on
I used a trick with adding numbers before an index term. It works, but I wouldn't recommend it for general use. It might be interesting to get an index within some group without sorting index entries within the letter/page group (it equals to an appearance in text, or sorting by page numbers without additional sorting key).
Handling numbers
Xindy is handling arabic and roman (I'm not 100% sure here) numbers well, see e.g. this question. There should be an addition to hexadecimal numbers. It could be solved with regular expressions in Lua. We may like to switch the output of the numbers (Western or Eastern Arabic, Roman, Urdu variant, Bengali, Indian numbers, ...).
Easy setting of the tags at the output side
We could be able to easily set up all tags used in the index. We could be able to limit number of elements, e.g. to see only the first element (first occurence of the index entry / definition / abbreviation). It's been discussed at TeX.SX.
Automatic repetion of the group letter
Once we have an index with more columns/pages, it would be nice to have group letters at the beginning of each column/page automatically repeated.
Adding text before and after index
We can do a lot of formatting at a TeX level, see this question, but there should be an easier way how to add information to the index. Moreover, we should be able to hyperlink whole index entry (see this question).
Creating more indices
We can do that easily with the help of the imakeidx package, see this example. We may like to have minindices (per chapter, section, page number), something similar to the minitoc package for Mini Table of Contents handling.
Index creation without TeXing
There are certain situations where we might not even need TeX at all. Lua could parse input TeX files. Michal.h21 showed one of the situations, indexing against keyword file, but that's a rare task.
Where the TeX world stands now?
Xindy
A new language can be set via xindy-make-rules for Xindy. The structure of data looks like this. We use or we leave empty so-called slots (arrays in arrays in an array).
$language = "Czech";
$prefix = "cs";
$script = "latin";
$alphabet = [
['A', ['a','A'], ['á','Á'],['ä','Ä']],
[], # a with breve (romanian)
[], # a with circumflex (romanian)
[], # a with ogonek (polish)
['B', ['b','B']],
[], # b with hook (hausa)
[... omitted...]
['Z', ['z','Z']],
[], # zh (albanian)
[], # zs (hungarian)
['Ž', ['ž','Ž']],
[], # z with acute (lower sorbian, polish)
[], # z with dot above (polish)
];
$ligatures = [
[['ß'], 'after', [['s','s']]],
];
@special = ('?', '!', '.', 'letters', '-', '\'');
$sortcase = "Aa";
#$sortcase = "aA";
do 'make-rules.pl';
TECkit
TECkit offers fast and flexible way for transliteration schemes, but it can be used only in XeTeX on-the-fly.
LuaTeX and ConTeXt
Hans Hagen and comp. are developing tools very fast (ConTeXt, LuaTeX). For instance, in the /texmf-dist/tex/context/base/sort-lan.lua file, we can find sorting rules for the Czech language:
[... omitted ...]
definitions["cz"] = {
replacements = {
{ "ch", ch }, { "Ch", ch }, { "CH", ch }
},
entries = {
["a"] = "a", ["á"] = "a", ["b"] = "b", ["c"] = "c", ["č"] = "č",
["d"] = "d", ["ď"] = "d", ["e"] = "e", ["é"] = "e", ["ě"] = "e",
["f"] = "f", ["g"] = "g", ["h"] = "h", [ch] = "ch", ["i"] = "i",
["í"] = "i", ["j"] = "j", ["k"] = "k", ["l"] = "l", ["m"] = "m",
["n"] = "n", ["ň"] = "n", ["o"] = "o", ["ó"] = "o", ["p"] = "p",
["q"] = "q", ["r"] = "r", ["ř"] = "ř", ["s"] = "s", ["š"] = "š",
["t"] = "t", ["ť"] = "t", ["u"] = "u", ["ú"] = "u", ["ů"] = "u",
["v"] = "v", ["w"] = "w", ["x"] = "x", ["y"] = "y", ["ý"] = "y",
["z"] = "z", ["ž"] = "ž",
},
orders = {
"a", "á", "b", "c", "č", "d", "ď", "e", "é", "ě",
"f", "g", "h", ch, "i", "í", "j", "k", "l", "m",
"n", "ň", "o", "ó", "p", "q", "r", "ř", "s", "š",
"t", "ť", "u", "ú", "ů", "v", "w", "x", "y", "ý",
"z", "ž",
},
upper = {
ch = CH,
},
lower = {
CH = ch,
}
}
definitions["cs"] = { parent = "cz" }
[... omitted ...]
It's a nice way of handling Lua tables, but I feel it could be simplified for a regular user for setting up own sorting/indexing rules.
CLDR: A way to go?
First look at the CLDR data
I like an approach by the Unicode Consortium (it's similar, maybe even related, to the biber tool for sorting bibliography entries), they are providing so called CLDR data (Unicode Common Locale Data Repository; I've found this project just recently), which includes data for index creation. Let me show you an example for the Czech language (after downloading and unzipping the core.zip file, http://unicode.org/Public/cldr/27/). The rules are written in the /code/common/main/ folder in the cs.xml file.
[... omitted ...]
<characters>
<exemplarCharacters>[a á b c č d ď e é ě f g h {ch} i í j k l m n ň o ó p q r ř s š t ť u ú ů v w x y ý z ž]</exemplarCharacters>
<exemplarCharacters type="auxiliary">[à ă â å ä ã ā æ ç è ĕ ê ë ē ì ĭ î ï ī ľ ł ñ ò ŏ ô ö ø ō œ ŕ ù ŭ û ü ū ÿ]</exemplarCharacters>
<exemplarCharacters type="index" draft="contributed">[A B C Č D E F G H {CH} I J K L M N O P Q R Ř S Š T U V W X Y Z Ž]</exemplarCharacters>
<exemplarCharacters type="punctuation">[\- ‐ – , ; \: ! ? . … ‘ ‚ “ „ ( ) \[ \] § @ * / \&]</exemplarCharacters>
[... omitted ...]
If we could parse this data (see this webpage about LDML; the Unicode Locale Data Markup Language), rules for a new language (Czech one in this example) can be set up. It's covering many languages and dialects, but e.g. Pohnpeian language is not covered, yet. Therefore I think, there still should be an easy way how to define a new language even with this data available.
There is probably a lot of tools to parse and preprocess these files, let me mention cldr from the JavaScript world.
It's hard to say if biber can be used for sorting in index creation, even if it could, it isn't handling typical issues of the index creation.
Notes about Sinhala and Tamil
Some years ago, Zdeněk Wagner from Prague, showed me his experiment in Devanāgarī, but it isn't included among official Xindy files, yet. Zdeněk told me his files are not completely ready to be published. So support for Brāhmī scripts is likely still weak.
One day, I would like to see sorted lists / indices e.g. in Pāḷi, preferably in Sinhala and Tamil not mentioning Japanese, Chinese, Korean and many more.
It looks it might be possible as CLDR contains files si.xml (ISO 639-1; Sinhala) and ta.xml (ISO 639-1; Tamil). Allkeys from Unicode might provide some help in certain languages/situations.
It looks that difficult part in programming is handling complex ligatures in those Eastern languages (next to handling radicals and number of strokes; that's another big issue). This is a portion of the table scheme a colleague of mine, Jan Kučera from Prague, sent me for Tamil. After we type a character followed by one of the MarkPlaceholders we are getting a ligature. That's a general idea, he has probably implemented it somewhere.
"", "^");
"அ", "a");
"ஆ", "ā", "A", "aa");
"இ", "i");
"ஈ", "ī", "I", "ee");
"உ", "u");
"ஊ", "ū", "U", "oo");
"எ", "e");
"ஏ", "ē", "E");
"ஐ", "ai");
"ஒ", "o");
"ஓ", "ō", "O");
"ஔ", "au", "ou", "ow");
"ஃ", "ḵ", "H", "q");
"க" + MarkPlaceholder, "k", "K", "kh", "G", "gh");
"ங" + MarkPlaceholder, "ṅ", "ng", "~g", "g"); // g is க in Baraha
"ச" + MarkPlaceholder, "c", "s", "ch", "C", "Ch");
"ஞ" + MarkPlaceholder, "ñ", "n^", "ny", "NY", "jn", "~j", "nJ"); // ந can be n^ in Madurai
"ட" + MarkPlaceholder, "ṭ", "T", "Th", "d", "D", "Dh"); // d is த in Baraha
"ண" + MarkPlaceholder, "ṇ", "N");
"த" + MarkPlaceholder, "t", "th", "dh");
"ந" + MarkPlaceholder, "n", "~n", "nN"); // n is ன in Baraha
"ப" + MarkPlaceholder, "p", "P", "ph", "b", "B", "bh");
"ம" + MarkPlaceholder, "m", "M");
"ய" + MarkPlaceholder, "y", "Y");
"ர" + MarkPlaceholder, "r");
"ல" + MarkPlaceholder, "l");
"வ" + MarkPlaceholder, "v", "w");
"ழ" + MarkPlaceholder, "z", "zh", "Lx");
"ள" + MarkPlaceholder, "ḷ", "L");
"ற" + MarkPlaceholder, "ṟ", "R", "rx");
"ன" + MarkPlaceholder, "ṉ", "_n", "n2");
"ஜ" + MarkPlaceholder, "j", "jh", "J");
"ஶ" + MarkPlaceholder, "ś");
"ஷ" + MarkPlaceholder, "ṣ", "sh"); // sh is ஶ in Baraha
"ஸ" + MarkPlaceholder, "S");
"ஹ" + MarkPlaceholder, "h", "~h");
"", "a", "^");
"ா", "ā", "A", "aa");
"ி", "i");
"ீ", "ī", "I", "ee");
"ு", "u");
"ூ", "ū", "U", "oo");
"ெ", "e");
"ே", "ē", "E");
"ை", "ai");
"ொ", "o");
"ோ", "ō", "O");
"ௌ", "au", "ou", "ow");
A closing note about Kana
Once, I tried to prepare a style for Xindy for Kana (Hiragana and Katakana), it worked, but the core which is to be processed are the kanjis. I would like to finish my notes with a preview of this Xindy experiment from 2010. I believe that this is not the easiest way how to set up and maintain definition files for index generation. These files are not ready for production, it's highly unlikely to have an index of just Kana. In case we have a couple of terms in Kana, we would probably use \index{sayounara@さようなら} in our document.
Thus, I was thinking in this post, if I(we) could do something about that to improve the current state in the world of indexing.
\labeland\refto apply page numbers (but that means multiple\labels for any term that is to be "indexed" more than once). since ordering is, to a great extent, subjective, i don't see an easy solution to this.\makeindexfrom this package is missing here?