I'm just starting to experiment with indexes, not very successfully yet. I am getting lots of duplicate entries, I suppose because of hyphenation. See my example below, which I've made from a grep of my real file, in which, for "Artha-śāstra", I particularly made sure that I took the first occurence of it and copied it over the later ones, the file having been written by two authors with the theoretical possibility of different unicode characters being used for the diacritical combinations. Looking at the idx file it seems that the two instances in which the discretionary hyphen is actually used it gets expanded to \indexentry{Artha\discretionary {-}{}{}śāstra@\textit {Artha\discretionary {-}{}{}śāstra}}{1} whereas in the other ones it is just \indexentry{Artha\-śāstra@\textit {Artha\-śāstra}}{1}. But maybe it's something else.
\documentclass{article}
\usepackage{fontspec}
\usepackage{makeidx}
\makeindex
\newcommand{\ttl}[1]{{\textit{#1}\index{#1@\textit{#1}}}}
\newcommand{\skt}[1]{{\textit{#1}}}
\begin{document}
Beginning at the top of the first passage, the first thing we shall comment on is the prevalence of magical pills in early tantric literature. These appear to be absent in the \ttl{Atharva\-veda\-pariśiṣṭa}, but we find the preparation of pills for making others fall unconscious in the mantra and \skt{prayoga} given in \ttl{Artha\-śāstra} 14.3.19ff. To call them pills, however, is potentially confusing, since in our tantric sources they are clearly to be popped into the \skt{sādhaka}’s mouth and held there, not swallowed, and they enable the \skt{sādhaka} to obtain powers.
\footnote{This is of course not something that develops for the first time in tantric sources: the \ttl{Artha\-śāstra} uses the magical power of the cremation ground in 14.3.29, 14.3.32 and 14.3.49. We may note however that it refers to it with the expression \skt{asaṃkīrṇa ādahane}, ‘in an unfrequented burning ground’. Tantric sources never, as far as we are aware, use this expression: they refer instead to the \skt{śmaśāna}.}
in \ttl{Atharva\-veda\-pari\-śiṣṭa} 36.21.1 and the fourteenth day of the dark fortnight (never the eighth) is specified for several of the magical \skt{prayoga}\/s given in the \ttl{Artha\-śāstra}.%
\footnote{\ttl{Artha\-śāstra} 14.3.28, 14.3.41, 14.3.49, 14.3.58, 14.3. 69, 14.3.70, 14.3.85. The fourteenth day is recommended for summoning the ancestors (\skt{pitṛ}) in order to satisfy them in \ttl{Atharva\-veda\-pari\-śiṣṭa} 43.6.1.}
in the \ttl{Artha\-śāstra} 4.5.7, and in \ttl{Aṣṭādhyāyī} 4.3.72.%
Probably widespread throughout the world is the use of effigies for damaging, destroying or gaining power over other people by a form of sympathetic magic in which whatever is done to the doll is held to be effected on the person represented. It is therefore no surprise to find them in, for example, the \ttl{Artha\-śāstra} (14.3.69–72):
\ttl{Rāmāyaṇa} 5.46.38, 6.45.22; \ttl{Artha\-śāstra} 1.19.11ab, 14.3.31, 14.3.45ab, 14.3.58; \ttl{Mahābhārata} 3.274.25, 7.58.9, 7.121.30, etc.
\printindex
\end{document}
Update:
The issue reported above occured when I generated the ind file with makeindex duplicates.idx. In the meantime I found out that the same file builds fine, without duplicates, when I generate it with the following command: xindy -M texindy duplicates.idx, thus the problem is solved for me. Curiously though,texindy duplicates.idx, which I would have expected to do the same thing, while taking care of different discretionary hyphens, does not manage to avoid duplicates for words containing discretionary hyphens in footnotes.
