# Importing phonetic characters from a .txt file fails with inputenc

I'm trying to import a dictionary I made in Toolbox (90's linguists software). I currently have the dictionary in a .txt file that LaTeX should be able to reproduce with a style file. In the .txt file I define different parts of each entry of the dictionary: the lexeme (\LX), the English translation (\GE), the phonetic transcription (\PH), etc.

The font used in the .txt file for the phonetic transcription should be imported directly to LaTeX. The line in the .sty file that does that is possibly the following:

\newcommand\PH[1]{\textipa{#1}}


My .tex:

\documentclass[12pt]{article}
\usepackage[a4paper, margin=1.3cm, twocolumn, columnsep=.3cm, driver=dvips]{geometry}
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{dict}%use the style file
\usepackage{paralist}%for the \compactdesc environment
\usepackage{tgtermes}
\setdefaultleftmargin{1em}{}{}{}{}{}%for illustrations
\usepackage{pstricks}
\psset{arrows=c-c}
\usepackage{overpic}
\usepackage{graphicx}
\newlength\cus
\sloppy

\begin{document}

\setlength\cus{0.08\linewidth}
\psset{linewidth=.5pt}

\begin{compactdesc}

\input{texdictkjempebra.txt}% source file
\end{compactdesc}

\end{document}


I guess also inputenc should take care of importing my weird characters.

Here's some example lines from my .txt dictionary (which is saved in UTF8):

\LX{ahat} \ph ahát̚ \PS{n} \GE{Sunday} \GN{hari mingu} \bw < Arab.

\LX{ajari} \ph adɟári \PS{v} \GE{to teach} \GN{mengajar}

\LX{bol} \ph bɔl \PS{n} \GE{mouth} \GN{mulut} \NT{wl 6}

\LX{bolkul} \ph bɔlkúl \PS{n} \GE{lip} \GN{bibir} \NT{wl 7}


The stuff after \ph is not imported correctly:

I get the following error codes (even though the .pdf is created):

• Package inputenc Error: Unicode char (U+FEFF)(inputenc) not set up for use with LaTeX.
• Undefined control sequence. \LX{ahat} \ph a
• Package inputenc Error: Unicode char ́ (U+301)(inputenc) not set up for use with LaTeX. \LX{ahat} \ph ahá
• Package inputenc Error: Unicode char ̚ (U+31A)(inputenc) not set up for use with LaTeX. \LX{ahat} \ph ahát̚

I guess you need the style file to reproduce correctly. I don't think I can attach anything here, but the file (which I modified, you would at least need to activate the code for \ph) can be downloaded at http://www.zas.gwz-berlin.de/uploads/media/dict.sty.

So: How do I make LaTeX read and print my input file correctly? It seems that LaTeX can't read the accute accents (representing high tone) and the diacritic "corners". In order to solve the first error message, I tried adding \DeclareUnicodeCharacter{FEFF}{} to the preamble, but then I get the following errors:

• Undefined control sequence. \DeclareUnicodeCharacter
• Missing \begin{document}. \DeclareUnicodeCharacter{F
• You should use UTF-8 for all your files. – egreg Apr 16 '16 at 10:52
• @egreg: My .txt is UTF-8, and I changed my preamble slightly: \usepackage[T1]{fontenc} \usepackage[utf8]{inputenc} \usepackage{lmodern} However, a new error occurs: Package inputenc Error: Unicode char (U+FEFF)(inputenc) not set up for use with LaTeX. I'm lost. – Eline Apr 16 '16 at 14:57
• That message seems to be about a BOM (byte order mark); you might be able to get away with it by adding \DeclareUnicodeCharacter{FEFF}{} in the preamble of the main TeX file. – egreg Apr 16 '16 at 15:25
• @egreg now I get Undefined control sequence. \DeclareUnicodeCharacter and Missing \begin{document}. \DeclareUnicodeCharacter{F. The characters that don't make it to the pdf are characters carrying tone, or sometimes just the tone diacritic (high tone, looks like an accute accent), and the unreleased stop mark (a sort of "corner" diacritic). – Eline Apr 17 '16 at 15:18
• You need \usepackage[utf8]{inputenc} – egreg Apr 17 '16 at 15:28

[U+FEFF] is a nonprinted byte order mark ("BOM") placed at the top of unicode files by some software, and notably used by Microsoft as a Magic Character to distinguish a unicode file, but it's not a part of the standard "UTF-8" encoding.

As egreg stated, define in the preamble. I would place it right after the \inputenc statement: \DeclareUnicodeCharacter{FEFF}{}

Or, save your .tex files as standard UTF-8 (not "UTF-8 with BOM" [and not] "UTF-8 with Byte Order Mark."

AND, for every character that still generates an "inputenc error"...

! Package inputenc Error: Unicode char  (U+????)
(inputenc)                not set up for use with LaTeX.


...for those, you will need to also setup \DeclareUnicodeCharacter{?}{?} for each of the different instances, but be sure and place an appropriate replacement value, because with "BOM" [U+FEFF] you aren't actually replacing with a printable character, you are just eliminating the "Magic Character".

If you use \usepackage[utf8]{inputenc}, it helps TeX understand the sequence of (UTF-8) bytes in the input file as Unicode characters, instead of individual bytes. For example, in Unicode, ahát̚ is a sequence of six Unicode characters (codepoints):

So to repeat, the main thing that \usepackage[utf8]{inputenc} does is to help TeX interpret ahát̚ as a sequence of 6 Unicode characters, instead of as 8 individual bytes.

(Aside: With ahát̚ it could also be five codepoints instead of six, as Unicode contains a precomposed character á for U+00E1 LATIN SMALL LETTER A WITH ACUTE, but Unicode doesn't contain precomposed characters for every conceivable combination like "t with combining left angle above".)

But even with this help from inputenc, TeX by itself doesn't know all the Unicode stuff like what to do with combining characters. To some extent it can be taught what to do with a given Unicode character, which is what \DeclareUnicodeCharacter does: you can say things like “when you see the Unicode character FEFF, do nothing”, with \DeclareUnicodeCharacter{FEFF}{}. (BTW, this line needs to come after you've loaded inputenc.)

A typical TeX distribution does come with a fair number of such definitions; for example “when you see the Unicode character 1E56, typeset a P with a dot above” (given by the line \DeclareUnicodeCharacter{1E56}{\.P} in tex/latex/oberdiek/ix-utf8enc.dfu on TeX Live).

But really, not everything can be given such definitions; how in TeX do you give a definition for U+0301 COMBINING ACUTE ACCENT, amounting to “go back and find the previous character you already typeset, and put an accent over it in the right place”? (It is probably possible to do this; it would just be a hack.)

Instead, you should consider the Unicode-compatible XeTeX or LuaTeX engines: all you need to do (in most cases) is compile with xelatex or lualatex instead of latex or pdflatex. These are built to be aware of Unicode and treat each Unicode codepoint just like any other character: they see U+0301 COMBINING ACUTE ACCENT and just put out that character and the (modern, OpenType) font that you ask them to use will contain information on how this is to be displayed.