# Language-specific removal of diacritics from text and bibliography

I am using XeLaTeX and BibLateX to write papers in the humanities. I routinely use transcriptions of Chinese (pinyin), Korean (McCune-Reischauer), Japanese (Hepburn), Manchu (Norman) etc. In addition, there are occasional references and quotes in European languages that use diacritics, such as French.

Up until now, I've written all my Chinese transcriptions in pinyin with tone marks (´ acute and  grave accents, ˇ háček, and ¯ macron). However, many journals discourage the use of tone marks in the transcriptions. I'm therefore looking for a way to strip my text of the diacritics for Chinese. As I see it, I face two problems:

1. I want to strip the bibliography-references of the pinyin diacritics without having to reenter them in the .bib file again, and

2. I want to remove only the diacritics in the Chinese transcriptions, not the macrons in Japanese (e.g. Tōkyō), or the typographically identical accents in French (e.g. problème).

Is there a way to do this?

UPDATE:

This is my try with the code kindly provided by PLK below. Unfortunately, it doesn't work for me:

\documentclass[utf8,12pt,letterpaper]{article}
\usepackage{setspace}
\usepackage[fallback]{xeCJK}
\usepackage{fontspec}
\defaultfontfeatures{Ligatures=TeX}
\setmainfont{Linux Libertine O}
\newcommand{\mainfontCJK}[0]{HanaMinA}
\setCJKmainfont[Scale=0.9]{\mainfontCJK}
\setCJKfallbackfamilyfont{\CJKrmdefault}[Scale=0.9]{NanumMyeongjo}
\XeTeXlinebreaklocale "zh"
\XeTeXlinebreakskip = 0pt plus 1pt
\usepackage[notes,strict,isbn=false,backend=biber,bibencoding=utf8,hyperref=true]{biblatex-chicago}

\DeclareSourcemap{
\maps[datatype=bibtex]{
\map{
\step[fieldsource=author, match=\regexp{[\x{300}\x{301}\x{304}\x{30c}]}, replace={}]
}
}
}
\DeclareNosort{
\nosort{type_names}{\regexp{[\x{300}\x{301}\x{304}\x{30c}]}}
}
\DeclareNoinit{
\noinit{\regexp{[\x{300}\x{301}\x{304}\x{30c}]}}
}

\begin{document}

\cite{WeiYuan18422004ShengwuJi}

\printbibliography

\end{document}


The citation comes out as usual, with the diacritics (please ignore the strange formatting of a name, I usually use something else to fix that, which I haven't included in this minimal example):

UPDATE

The bibliography entry used in the example above:

@book{WeiYuan18422004ShengwuJi,
author = {Wèi, 魏源, Yuán},
title = {Shèngwǔjì},
usere = {Record of the sacred wars},
publisher = {Yuèlù shūshè},
date = {2004},
origdate = {1842},
edition = {\autocap{c}ritical edition},
maintitle = {Wèi Yuán quánjí},
volume = {3}
}

• My first thought would be to do this with a script and not so much with TeX. Change ō with o and similar can be quite easily accomplished. Bit more a question for stackoverflow in that regard though. – Mythio Apr 23 '13 at 14:31
• You should use e.g. {\^o} in your bibliography anyway, perhaps the simplest would be to (re)define those accents to just give their arguments. No need to mangle the bibliography that way. For the text itself, I'd create a script for sed (if on a Unixy system) to replace the UTF-8 accented characters. Doing the replacement only for Chinese transliterations is extremely hard, unless marked by something like \foreignlanguage{chinese}{transliteration here}. In the later case it is only tricky. – vonbrand Apr 23 '13 at 17:36

If you are using biber with biblatex (and you probably should be if you need UTF-8 support like this), then this is possible with the biber mapping feature. For example, to strip acutes, graves, macron and hacek from authors, put this in your preamble:

\DeclareSourcemap{
\maps[datatype=bibtex]{
\map{
\step[fieldsource=author, match=\regexp{[\x{300}\x{301}\x{304}\x{30c}]}, replace={}]
}
}
}


Note that this strips them dynamically whilst reading the .bib and so your bib is not modified.

Also, you may want to strip these for sorting and/or generating initials in which case you also add to your preamble:

\DeclareNosort{
\nosort{type_names}{\regexp{[\x{300}\x{301}\x{304}\x{30c}]}}
}
\DeclareNoinit{
\noinit{\regexp{[\x{300}\x{301}\x{304}\x{30c}]}}
}


The issue of only stripping Chinese should be handled by the .bib having some way of telling Chinese entries from others - perhaps a HYPHENATION or LANGUAGE field. Then the first code snippet above becomes something like:

\DeclareSourcemap{
\maps[datatype=bibtex]{
\map{
\step[fieldsource=language, match=chinese, final]
\step[fieldsource=author, match=\regexp{[\x{300}\x{301}\x{304}\x{30c}]}, replace={}]
}
}
}


Of course, this all assumes you are encoding your .bib in UTF-8 (which you can with biber). If you are using TeX macros, then the regexps would be a bit more complex ...

An alternative, using Unicode classes is to remove all Unicode MARK graphemes instead of individual graphemes:

\step[fieldsource=author, match=\regexp{\p{Mark}}, replace={}]

• Thanks a lot! It didn't work for me, I added my example with your code to the question, so as to provide a minimal example. What am I doing wrong? – Mårten Apr 23 '13 at 21:17
• Can you add the bibliography file (just that one entry is fine) to the example? I need that to test. – PLK Apr 24 '13 at 5:43
• Sure, it's added! – Mårten Apr 24 '13 at 13:19
• I see the problem. It's something I've been meaning to look at in biber`. I am making the NFD/NFC Unicode boundaries of the code more robust. Nearly done and will update the 1.6 dev version soon. I have your example working now. – PLK Apr 25 '13 at 8:10
• Great, thank you so much. Will you post here when the update is done? – Mårten Apr 25 '13 at 13:49