Longer post
I can confirm that, for instance we can compare http://vietunicode.sourceforge.net/charset/quytacABC_en.html written by a native Vietnamese and a listing of alphabets from xindy, http://xindy.sourceforge.net/doc/make-rules-alphabets-doc.pdf, page 41, there are differences as stated in the question.
I believe that the reason for this error is that the Western European languages (general style in xindy) don't recognize all the tonal diacritics. Those we know are sorted and those two unrecognized marks were put at the end of a sorting order. These things happen.
How to fix it? Just editing vietnamese/utf8.pl.in is not enough, it is a Perl script to generate the necessary files, especially the xdy files. I strongly don't recommend to edit this file directly as it is an official version and can be automatically overwritten with a new version of xindy when TeXLive/MikTeX is updated.
This is what I did to fix it locally in my working directory:
- I downloaded the latest version of
xindy-make-rules from http://sourceforge.net/projects/xindy/files/xindy-make-rules/0.2/ to my working directory.
- I unzipped the file.
- I made a backup by copying
alphabets/vietnamese directory to a new folder named alphabets/vietnamesemal. I entered that new folder.
- There is just one file. I made a copy of it and named it
utf8.pl-mal.in to keep the original version in the same folder, if needed later.
- I edited this file in
gEdit, http://wiki.gnome.org/Apps/Gedit. It is one of many editors capable of handling full set of UTF-8 characters.
- I rearranged the characters according to the Vietnamese sorting rules. This is the new file.
#!/usr/bin/perl
$language = "Vietnamese";
$prefix = "vi";
$script = "latin";
$alphabet = [
['A', ['a','A'],['à','À'],['ả','Ả'],['ã','Ã'],['á','Á'],['ạ','Ạ']],
['Ă', ['ă','Ă'],['ằ','Ằ'],['ẳ','Ẳ'],['ẵ','Ẵ'],['ắ','Ắ'],['ặ','Ặ']],
['Â', ['â','Â'],['ầ','Ầ'],['ẩ','Ẩ'],['ẫ','Ẫ'],['ấ','Ấ'],['ậ','Ậ']],
[], # a with ogonek (polish)
['B', ['b','B']],
[], # b with hook (hausa)
['C', ['c','C']],
[], # ch (spanish/traditonal)
[], # cs (hungarian)
[], # c with caron (many)
[], # c with acute (croatian, lower sorbian, polish)
[], # c with circumflex (esperanto)
[], # c with cedilla (albanian, kurdish, turkish)
['D', ['d','D']],
[], # dh (albanian)
[], # dz (hungarian)
[], # dzs (hungarian)
[], # d+z with caron (croatian)
[], # d+z with acute (upper sorbian)
[], # d with caron (slovak/large)
['Đ', ['đ','Đ']],
[], # d with hook (hausa)
[], # eth (icelandic)
['E', ['e','E'],['è','È'],['ẻ','Ẻ'],['ẽ','Ẽ'],['é','É'],['ẹ','Ẹ']],
[], # e with caron (lower/upper sorbian)
['Ê', ['ê','Ê'],['ề','Ề'],['ể','Ể'],['ễ','Ễ'],['ế','Ế'],['ệ','Ệ']],
[], # e with diaeresis (albanian)
[], # e with ogonek (polish)
['F', ['f','F']],
['G', ['g','G']],
[], # gj (albanian)
[], # gy (hungarian)
[], # g with circumflex (esperanto)
[], # g with breve (turkish)
[], # g with cedilla/comma (latvian)
[], # postpalatal fricative (gypsy/northrussian)
['H', ['h','H']],
[], # h with circumflex (esperanto)
[], # ch (many)
[], # dotless i (turkish)
['I', ['i','I'],['ì','Ì'],['ỉ','Ỉ'],['ĩ','Ĩ'],['í','Í'],['ị','Ị']],
[], # i with inverted breve below (gypsy/northrussian)
[], # i with circumflex (kurdish, romanian)
[], # i with diaeresis (gypsy/northrussian)
['J', ['j','J']],
[], # j with circumflex (esperanto)
['K', ['k','K']],
[], # kh (gypsy/northrussian)
[], # k with cedilla/comma (latvian)
[], # k with hook (hausa)
[], # x (gypsy/northrussian)
[], # l with stroke (lower/upper sorbian)
['L', ['l','L']],
[], # lj (croatian)
[], # ll (albanian, spanish/traditonal)
[], # ly (hungarian)
[], # l with cedilla/comma (latvian)
[], # l with stroke (polish)
[], # l with caron (slovak/large)
['M', ['m','M']],
['N', ['n','N']],
[], # nj (albanian, croatian)
[], # ny (hungarian)
[], # n with caron (slovak/large)
[], # n with acute (lower/upper sorbian, polish)
[], # n with tilde (spanish/modern, spanish/traditional)
[], # n with cedilla/comma (latvian)
['O', ['o','O'],['ò','Ò'],['ỏ','Ỏ'],['õ','Õ'],['ó','Ó'],['ọ','Ọ']],
[], # o with acute (polish, upper sorbian)
['Ô', ['ô','Ô'],['ồ','Ồ'],['ổ','Ổ'],['ỗ','Ỗ'],['ố','Ố'],['ộ','Ộ']],
['Ơ', ['ơ','Ơ'],['ờ','Ờ'],['ở','Ở'],['ỡ','Ỡ'],['ớ','Ớ'],['ợ','Ợ']],
[], # o with diaeresis (hungarian, turkish)
['P', ['p','P']],
[], # ph (gypsy/northrussian)
['Q', ['q','Q']],
['R', ['r','R']],
[], # rr (albanian)
[], # r with caron (czech, slovak/large, upper sorbian)
[], # r with acute (lower sorbian)
[], # r with cedilla/comma (latvian)
['S', ['s','S']],
[], # sh (albanian)
[], # sz (hungarian)
[], # s with caron (many)
[], # s with acute (lower sorbian, polish)
[], # s with circumflex (esperanto)
[], # s with comma below (romanian)
[], # s with cedilla (kurdish, turkish)
[], # z (estonian)
[], # z with caron (estonian)
['T', ['t','T']],
[], # th (albanian)
[], # ty (hungarian)
[], # t with caron (slovak/large)
[], # t with comma below (romanian)
[], # c with acute (upper sorbian) @@
['U', ['u','U'],['ù','Ù'],['ủ','Ủ'],['ũ','Ũ'],['ú','Ú'],['ụ','Ụ']],
[], # u with breve (esperanto)
[], # u with circumflex (kurdish)
['Ư', ['ư','Ư'],['ừ','Ừ'],['ử','Ử'],['ữ','Ữ'],['ứ','Ứ'],['ự','Ự']],
[], # u with diaeresis (hungarian, turkish)
['V', ['v','V']],
['W', ['w','W']],
[], # o with tilde (estonian)
[], # a with diaeresis (estonian)
[], # o with diaeresis (estonian)
[], # u with diaeresis (estonian)
['X', ['x','X']],
[], # xh (albanian)
['Y', ['y','Y'],['ỳ','Ỳ'],['ỷ','Ỷ'],['ỹ','Ỹ'],['ý','Ý'],['ỵ','Ỵ']],
[], # y preceded by apostrophe (hausa)
[], # yogh (english)
['Z', ['z','Z']],
[], # zh (albanian)
[], # zs (hungarian)
[], # z with caron (many)
[], # z with acute (lower sorbian, polish)
[], # z with dot above (polish)
[], # thorn (icelandic)
[], # wynn (english)
[], # ligature ae (danish, icelandic, norwegian)
[], # o with stroke (danish, norwegian)
[], # a with ring above (danish, norwegian, swedish)
[], # a with diaeresis (finnish, swedish)
[], # o with diaeresis (finnish, swedish)
[], # a with ring above (icelandic)
];
$sortcase = 'Aa';
#$sortcase = 'aA';
$ligatures = [
];
@special = ('?', '!', '.', 'letters', '-', '\'');
do 'make-rules.pl';
- I entered
alphabets/ directory by cd ..
- Then I run
perl vietnamesemal/utf8.pl-mal.in vietnamesemal/vietnamesemal. I have got a message that Alphabet has 126 elements. I have received several new files there.
I entered vietnamesemal again. I changed the first line in vietnamesemal-test.xdy to (require "vietnamesemal.xdy"). It means we won't need subdirectories to our work later.
I modified vietnamesemal-doc.tex a bit: \subsection -> \section; \subsubsection to \subsection (four times); I commented out \idef\fdef (twice) and \newpage (once). This is the modified version of that file.
\section{Vietnamese}
\subsection{Alphabet}
%\icod\fcod
\begin{alphabet}
a\,A & à\,À & ả\,Ả & ã\,Ã & á\,Á & ạ\,Ạ\\
ă\,Ă & ằ\,Ằ & ẳ\,Ẳ & ẵ\,Ẵ & ắ\,Ắ & ặ\,Ặ\\
â\, & ầ\,Ầ & ẩ\,Ẩ & ẫ\,Ẫ & ấ\,Ấ & ậ\,Ậ\\
b\,B\\
c\,C\\
d\,D\\
đ\,Đ\\
e\,E & è\,È & ẻ\,Ẻ & ẽ\,Ẽ & é\,É & ẹ\,Ẹ\\
ê\,Ê & ề\,Ề & ể\,Ể & ễ\,Ễ & ế\,Ế & ệ\,Ệ\\
f\,F\\
g\,G\\
h\,H\\
i\,I & ì\,Ì & ỉ\,Ỉ & ĩ\,Ĩ & í\,Í & ị\,Ị\\
j\,J\\
k\,K\\
l\,L\\
m\,M\\
n\,N\\
o\,O & ò\,Ò & ỏ\,Ỏ & õ\,Õ & ó\,Ó & ọ\,Ọ\\
ô\,Ô & ồ\,Ồ & ổ\,Ổ & ỗ\,Ỗ & ố\,Ố & ộ\,Ộ\\
ơ\,Ơ & ờ\,Ờ & ở\,Ở & ỡ\,Ỡ & ớ\,Ớ & ợ\,Ợ\\
p\,P\\
q\,Q\\
r\,R\\
s\,S\\
t\,T\\
u\,U & ù\,Ù & ủ\,Ủ & ũ\,Ũ & ú\,Ú & ụ\,Ụ\\
ư\,Ư & ừ\,Ừ & ử\,Ử & ữ\,Ữ & ứ\,Ứ & ự\,Ự\\
v\,V\\
w\,W\\
x\,X\\
y\,Y & ỳ\,Ỳ & ỷ\,Ỷ & ỹ\,Ỹ & ý\,Ý & ỵ\,Ỵ\\
z\,Z
\end{alphabet}
%\idef\fdef
\subsection{Ligatures}
\begin{flushleft}
None.
\end{flushleft}
\subsection{Upper-/lowercase words}
Capitalized or uppercase words are sorted \emph{before} otherwise equal lowercase words.
\subsection{Special characters}
The order of special characters and letters is:
\begin{flushleft}
?\hspace{4mm}!\hspace{4mm}.\hspace{4mm}letters\hspace{4mm}-\hspace{4mm}'
\end{flushleft}
%\newpage
- I went up one level and I have modified the
alphabets-doc.tex file quite a lot to this form. I ran it through lualatex which uses Latin Modern typefaces (its default font family) and it assures that all the tone marks will be typeset properly. It would be easy to get xelatex working on that file.
I enclose the new file and a preview of it. We can check if everything is all right, it should be now after those changes.
%! lualatex alphabets-doc.tex
\documentclass[a4paper]{article}
\pagestyle{empty}
%\usepackage{a4wide}
\usepackage{luatextra}
%\usepackage[TS1,LGR,T2A,T1]{fontenc}
%\usepackage[colorlinks]{hyperref}
%\usepackage[latin2,cp1251,cp1252]{inputenc}
%\newcommand{\idef}{\inputencoding{cp1252}}
%\newcommand{\fdef}{\fontencoding{T1}\selectfont}
%\newcommand{\icod}{\inputencoding{cp1252}}
%\newcommand{\fcod}{\fontencoding{T1}\selectfont}
%\newcommand{\ienc}[1]{\renewcommand{\icod}{\inputencoding{#1}}}
%\newcommand{\fenc}[1]{\renewcommand{\fcod}{\fontencoding{#1}\selectfont}}
\newenvironment{alphabet}{\begin{tabular}{*{16}{l}}%
% &
% \small (\v{}) & \small (\'{}) & \small (\`{}) & \small (\u{}) &
% \small (\^{}) & \small (\~{}) & \small (\r{}) & \small (\"{}) &
% \small (,) & \small (\c{}) & \small (k{}) & \small (\.{}) &
% \small (-) & \small (\={}) & \small (?)\\
}{\end{tabular}}
\setlength{\topskip}{0mm}
\setlength{\topmargin}{-15mm}
\setlength{\textheight}{260mm}
\setcounter{tocdepth}{2}
\begin{document}
\title{Alphabets}
\author{Generated by \tt make-rules.pl}
%\maketitle
\begin{center}
{\LARGE Alphabets} --
\texttt{\Large Generated by \tt make-rules.pl} \par\medskip
\large \today
\end{center}
%\tableofcontents
%\newpage
%\input{alphabets-inc.tex}
\input{vietnamesemal/vietnamesemal-doc.tex}
\end{document}
In the vietnemasemal folder I created a new TeX file named mal-vietnamese.tex to test it. I have used several resources to get a longer index. The links are mentioned in the TeX file.
I have also added word groups by creating malstyle.xdy file. It is a part of the mal-vietnamese.tex file. It uses one command named \malgroup and we can control its style within the TeX file directly without an additional run of xindy.
This is the file I mentioned:
%! lualatex mal-vietnamese.tex
\documentclass[a4paper]{article}
\pagestyle{empty}
\usepackage{multicol}
\usepackage{luatextra}
\usepackage{makeidx}
\makeindex
% Rather new tool... % makeindex, xindy
%\usepackage[xindy]{imakeidx}
%\makeindex[columns=3, options={-M texindy -M vietnamesemal -M malstyle}]
% The sources of data...
% http://tex.stackexchange.com/questions/142345/sorting-vietnamese-utf8-index-with-make-rules-in-xindy-package
% http://vietunicode.sourceforge.net/charset/quytacABC_en.html
% http://tex.stackexchange.com/questions/131334/how-to-sort-alphabet-index-in-vietnamese-correctly
\usepackage{filecontents}
\begin{document}
\begin{filecontents*}{malstyle.xdy}
;; malstyle.xdy
(markup-letter-group :open-head "~n\malgroup{ " :close-head " }" :capitalize)
\end{filecontents*}
\def\malgroup#1{\par\medskip\textbf{\large$\sim$#1$\sim$}\par\nopagebreak}
\begingroup
\def\thispagestyle#1{}
\printindex
\endgroup
\index{Hiên}
\index{Hiền}
\index{Hiển}
\index{Hiễn}
\index{Hiến}
\index{Hiện}
\index{Lản}
\index{Lãn}
\index{Lán}
\index{Lạn}
\index{Apple}\index{Apricot}\index{Avocado}\index{Banana}
\index{Bilberry}\index{Blackberry}\index{Blackcurrant}\index{Blueberry}
\index{Currant}\index{Cherry}\index{Cherimoya}\index{Clementine}
\index{Date}\index{Damson}\index{Dragonfruit}\index{Durian}
\index{Eggplant}\index{Elderberry}\index{Feijoa}\index{Gooseberry}
\index{Grape}\index{Grapefruit}\index{Guava}\index{Huckleberry}
\index{Jackfruit}\index{Jambul}\index{Kiwi fruit}\index{Kumquat}
\index{Legume}\index{Lemon}\index{Lime}\index{Lychee}\index{Mandarine}
\index{Mango}\index{Melon}\index{Nectarine}\index{Orange}\index{Peach}
\index{Pear}\index{Pitaya}\index{Physalis}\index{Plum}\index{Pineapple}
\index{Pomegranate}\index{Purple Mangosteen}\index{Raisin}\index{Raspberry}
\index{Rambutan}\index{Redcurrant}\index{Salal berry}\index{Satsuma}
\index{Star fruit}\index{Strawberry}\index{Tangerine}\index{Tomato}
\index{Ugli fruit}\index{Watermelon}\index{Ziziphus mauritiana}
\index{Đồng biến}
\index{Nghịch biến}
\index{Dao động điều hòa}
\index{Ếch}
Plain text\ldots
\end{document}
I ran these three commands and we are getting an index according to the Vietnamese sorting rules. I am submitting the utf8.pl-mal.in file to the authors of xindy, maybe they will use it in some new release of xindy. We will see! Enjoy and good bye! \index{tạm biệt}
lualatex mal-vietnamese.tex
xindy -M texindy -M vietnamesemal-test -M malstyle mal-vietnamese.idx
lualatex mal-vietnamese.tex
A working example from scratch without changing things
I cannot offer you a general solution how to set up a new version globally, I rather show you how it works in my working directory, you can probably manage the rest, if needed.
I have received first response from the xindy community and I have got a testing file (appreciations go to Hien Pham). Let me show you what I did:
- I created a bright new folder,
hien-example where I will be working in.
- I created a new TeX file there,
IndexVietnamese.tex. Because I cannot use texindy (my Windows home directory contains diacritical letters and I cannot change it without risk) and therefore I cannot use imakeidx at its best: both parameters xindy and texindy run texindy, I wil run this example from the command line.
- Please get
vietnamesemal-test.xdy from my server to that directory.
- Please get
vietnamesemal.xdy from my server as well.
- Run these four lines. If you wish to have a version without the group letters, delete
-M malstyle (twice):
lualatex --shell-escape IndexVietnamese.tex
xindy -M vietnamesemal-test -M malstyle authors.idx
xindy -M vietnamesemal-test -M malstyle IndexVietnamese.idx
lualatex --shell-escape IndexVietnamese.tex
I enclose those three files (one file in the post, two files are downloadable from my server) and a preview of page 3 (Author index) and pages 5 and 6 (Topic index).
This is the IndexVietnamese.tex file:
%! lualatex --shell-escape IndexVietnamese.tex
%! xindy -M vietnamesemal-test -M malstyle authors.idx
%! xindy -M vietnamesemal-test -M malstyle IndexVietnamese.idx
%! lualatex --shell-escape IndexVietnamese.tex
\documentclass{book}
%\usepackage[utf8]{vietnam}
\usepackage{luatextra}
\usepackage[noautomatic]{imakeidx} % [noautomatic]
\makeindex[title=Topic index]
% columns=3, options={-M vietnamesemal-test},
% from lang/vietnamese/utf8-lang to a testing version
\makeindex[name=authors, title=Author index]
% columns=3, options={-M vietnamesemal-test},
% from lang/vietnamese/utf8-lang to a testing version
\makeindex
\usepackage{natbib}
% \usepackage[noautomatic]{imakeidx}
% \makeindex[columns=3,options={-M lang/vietnamese/utf8-lang}]
%\renewcommand\indexname{Topic index}
\usepackage{filecontents}
% For purpose of the post...
\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhf{}
\renewcommand{\headrulewidth}{0pt}
\begin{document}
\begin{filecontents*}{malstyle.xdy}
;; malstyle.xdy
(markup-letter-group :open-head "~n\malgroup{ " :close-head " }" :capitalize)
\end{filecontents*}
\def\malgroup#1{\par\medskip\textbf{\large$\sim$#1$\sim$}\par\nopagebreak}
\citeindextrue
\chapter{Hello}
And it has fancy authors. \index[authors]{Doreian, Patrick} \index[authors]{Batagelj, Vladimir} \index[authors]{Ferligoj, Anuška}
key words: network\index{network}, blockmodeling\index{blockmodeling}, and so on.
% I am making this index shorter to fit onto two pages...
Some text.\index{Apple}\index{Apricot}\index{Avocado}%\index{Banana}
\index{Bilberry}\index{Blackberry}\index{Blackcurrant}%\index{Blueberry}
\index{Currant}\index{Cherry}\index{Cherimoya}%\index{Clementine}
\index{Date}\index{Damson}%\index{Dragonfruit}\index{Durian}
\index{Eggplant}\index{Elderberry}\index{Feijoa}%\index{Gooseberry}
\index{Grape}\index{Grapefruit}\index{Guava}\index{Huckleberry}
\index{Jackfruit}\index{Jambul}\index{Kiwi fruit}\index{Kumquat}
\index{Legume}\index{Lemon}\index{Lime}\index{Lychee}\index{Mandarine}
\index{Mango}\index{Melon}\index{Nectarine}\index{Orange}\index{Peach}
\index{Pear}\index{Pitaya}\index{Physalis}\index{Plum}\index{Pineapple}
\index{Pomegranate}\index{Purple Mangosteen}\index{Raisin}\index{Raspberry}
\index{Rambutan}\index{Redcurrant}\index{Salal berry}\index{Satsuma}
\index{Star fruit}\index{Strawberry}\index{Tangerine}\index{Tomato}
\index{Ugli fruit}\index{Watermelon}\index{Ziziphus mauritiana}
\index{Đồng biến}
\index{Ông bà}
\index{Ăn ở}
\index{Ương ngạnh}
\index{Nghịch biến}
\index{Dao động điều hòa}
\index{Ếch}
\index{Ẩm kế}
\index{Ấm áp}
\index{Ầm ĩ}
\index{Ậm ừ}
\index{Âm hưởng}
\index{Ẫm ờ}
\index{Ơn huệ}
\index{Lan}
\index{Làn}
\index{Lãn}
\index{Lạn}
\index{Lản}
\index{Lán}
\index{Ãn}\index{An}\index{Án}\index{Ạn}\index{Àn}\index{Ản}
\index{Hiện}\index{Hiển}\index{Hiến}\index{Hiên}\index{Hiễn}\index{Hiền}
\index{choảng}\index{choạng}\index{choang}\index{choáng}\index{choãng}\index{choàng}
\index{provinces!Ontario}
\index{provinces!Saskatchewan}
\index{provinces!British Columbia}
\index{provinces!Alberta!Edmonton}
\index{territories@\textbf{territories}|see{vùng đất}}
\index{vùng đất} % a correction
% For purpose of this post...
\begingroup
\def\thispagestyle#1{}
\printindex[authors]
\def\indexname{Topic index} % The imakeidx is not working for me at its best...
\printindex
\endgroup
\end{document}
This is the vietnamesemal-test.xdy file and finally this is the vietnamesemal.xdy file (both generated by xindy-make-rules). I tried my best to post them here, but the second generated file contains some characters which weren't displayed correctly here and therefore it is not working in xindy then, please download them, e.g. by:
wget http://striz7.fame.utb.cz/tex-sx/vietnamesemal-test.xdy
wget http://striz7.fame.utb.cz/tex-sx/vietnamesemal.xdy
I also post the TeX file (just in case) and the PDF file (if you would like to zoom in the pages), you can get the files by:
wget http://striz7.fame.utb.cz/tex-sx/IndexVietnamese.tex
wget http://striz7.fame.utb.cz/tex-sx/IndexVietnamese.pdf
