Unicode characters in pdflatex output using hexcode without UTF-8 input

Question

I want to use Unicode characters (for Korean language, i.e. Hangul: U+AC00 - U+D7A3; maybe also U+1100 - U+11FF) in the output of latex(/pdflatex) while the LaTeX source file should be coded in 7 bit ASCII as before.

I would like to have something similar to \ding{number} for ZapfDingbats for Unicode selected via its hexcode, something like:

\unicodechar{bc73}

to generate a character. The output should be generated as before:

latex -> dvips -> ps2pdfwr

Until now I have only seen unicode examples using UTF-8 encoding in source code - which is not an option for me - and typically using xelatex (disrupting my book - having problems with the packages I am using). I use TeXLive 2012 (Debian) fully installed (incl. CJK package) under Xubuntu 12.10, so in principle the fonts should be there.

I would really appreciate any answer or at least a hint to the solution - or in worst case a reason why this may not be possible at all.

NEWLY ADDED: Please read the facts above. No comment or answer is even near the question - look at unicode questions here - this question is totally different. No German Umlaute or Eurosign - that's simple. No entering of Unicode chars in the editor or elsewhere, only visible in the output. No XeLaTeX or LuaLaTeX.

Just an example to print Hangul characters under conditions given above or a reason why it won't work with latex/pdflatex - please.

Answer 1

Well in theory you can type your whole text in ascii, you only need to know which octets (8-bit-packets) to use:

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{textcomp}
\usepackage[utf8]{inputenc}

\begin{document}

Euro (€): ^^e2^^82^^ac

ä: ^^c3^^a4

\end{document}

The main problem is to get the utf8-hex-notation (e282ac) from the unicode name (U+20AC). In theory it can be calculated (and inputenc is doing it when processing \DeclareUnicodeCharacter{20AC}{\texteuro}) but I don't know a simple way to use the inputenc commands to get the values.

This input notation should work with the cjk-package (with utf8-option) too.

With the ucs-package (loaded when you use the option utf8x) you can use the command \unichar. But you should be aware that ucs can clash with some packages, e.g. it is mentioned in the biblatex list of incompatible packages.

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{textcomp}
\usepackage[utf8x]{inputenc}

\begin{document}
\unichar{"00E4}
\unichar{"20AC}
\end{document}

I have no idea if it would work together with cjk.

cjk offers also some text commands to enter symbols, e.g. \Li4 \chun1. but I don't know if this covers your symbols too.

Answer 2

I think enctex will allow you to do this.

First, write the unicode string in a \message{¾} (for example). Then compile your file, and you will see a multi-byte string on the output — ^^c2^^be in this example.

Now, in the top of your file (or, even better, in a format), type \mubyte \someTeXCommand ^^c2^^be\endmubyte and \mubytein=1.

Compile your file (or your format) with the enc option (it requires ini option too, and LaTeX may require the etex option on top), and all should be fine — hopefully!

I personally keep a list of such encodings in my format, for example to typeset mathematics in unicode or use special characters in text such as “…”

Answer 3

I just had a problem with this, and answers by @UlrikeFischer and @RD6137 are essentially correct, but I'll post a writeup here; although I'm not sure about the Hangul characters per se, I'll be focusing on "Unicode characters" and "hexcode" part of the question; all of this was tested with pdflatex.

No German Umlaute or Eurosign - that's simple.

Actually, not simple (Command \texteuro unavailable in encoding T1) - just the complexity was hidden by the textcomp package.

First, let's focus on input. If I do this in my terminal (bash shell on Linux):

$ echo -n € | hexdump -C
00000000  e2 82 ac                                          |...|
00000003

... I can see that what I entered as € in the shell, is actually a sequence of three bytes: 0xE2, 0x82, and 0xAC. Maybe it would be easier to also use my script utfinfo.pl:

$ echo -n € | perl utfinfo.pl 
Got 1 uchars
Char: '€' u: 8364 [0x20AC] b: 226,130,172 [0xE2,0x82,0xAC] n: EURO SIGN [Currency Symbols]

This tells us that the character/glyph for Euro sign € is filed as entry number 8364 (hex 0x20AC) in the Unicode table - and encoded (as per UTF-8, I guess) as the sequence of bytes 226,130,172 ( 0xE2,0x82,0xAC in hex ); exactly the same one which hexdump gave us earlier. Which brings us to this:

No entering of Unicode chars in the editor or elsewhere,

If this was a reference to @RD6137's \message{¾}, I believe that was just a suggestion for one method on how to obtain the actual byte sequence for a Unicode character. When you have a code like U+20AC, that is just the Unicode name - think of it as the integer index of the entry in the Unicode table. But that is not the actual byte sequence in UTF-8.

And the only think you can enter directly in a Latex source file are raw UTF-8 bytes, by using the double caret ^^ escape sequence (I think the two characters following this are interpreted as hexadecimal); so having the U+20AC Unicode index/name won't help you much - you need the actual UTF-8 byte sequence 0xE2,0x82,0xAC. (As mentioned earlier, there seem to be no conventient/easy Latex macros that automatically map from Unicode index to a UTF-8 byte sequence).

So by entering ^^e2^^82^^ac as ASCII text in the Latex source doc, Latex will parse this and interpret it as if you entered byte sequence 0xE2,0x82,0xAC - which would be the exact same result if you pasted the € character in a text editor, and saved the text file with UTF-8 encoding. However, if these three bytes are entered in a Latex doc, they will be interpreted as a single unicode character only if there is \usepackage[utf8]{inputenc}! Otherwise, Latex will map them to individual glyphs to whatever is its current font setting.

Which brings us to another problem - font encoding. Consider this MWE:

\documentclass{article}
% \usepackage[T1]{fontenc}
\usepackage{lmodern} % use font Latin Modern (lmodern / lmr)

\usepackage{fonttable}
\makeatletter
\def\myfontinfo{font: encoding \f@encoding, family: \f@family, series: \f@series, shape: \f@shape, size: \f@size, baselineskip: \f@baselineskip}
\makeatother

\begin{document}
  \typeout{\myfontinfo} % write to stdout
  \myfontinfo           % typeset in document
  , ^^e2^^82^^ac %
  \vspace*{-2em}
  \makeatletter
  %% \xfonttable{ encoding }{ family }{ series }{ shape }
  \xfonttable{\f@encoding}{\f@family}{\f@series}{\f@shape}
  \makeatother
\end{document}

If you compile this as-is, then Latex automatically assumes it's in OT1 encoding, which prints font table as on the image left (click for full size):

... and if you enable [T1]{fontenc} and recompile, Latex will use T1 encoding, generating a completely different font table - as on the middle image. You will note, since [utf8]{inputenc} is not specified, Latex interpreted ^^e2^^82^^ac as individual characters âĆň in both cases, exactly as per the tables in each case (reading note for font table: left and top ' are in octal; right and bottom " are in hex; so % which is 37 decimal, is '04x&'5='045 octal, and "2x&'5="25 hex; while - which is 45 decimal, is '05x&'5='055 octal, and "2x&"D="2D hex). However, you will also note that neither OT1 nor T1 contain the euro symbol - so where is it? It turns out, it is in another font encoding called TS1, and is on the rightmost image above (just use [TS1]{fontenc} in the MWE above), where it is glyph number 191 decimal ('277 octal, "BF" hex).

So, if we just add \usepackage[utf8]{inputenc} to the above example, it will now crash with:

! Package inputenc Error: Unicode char \u8:€ not set up for use with LaTeX.
See the inputenc package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                                  
l.15   , ^^e2^^82^^ac
                      %
? 

We can superficially remedy that by using \DeclareUnicodeCharacter - which now needs the Unicode index in hex, not the UTF-8 byte sequence. So, say you add \DeclareUnicodeCharacter{20AC}{TheEURO} right after the \usepackage[utf8]{inputenc}; you'll notice, now the sequence ^^e2^^82^^ac results with just the word TheEURO! So, \DeclareUnicodeCharacter can basically just call a macro for you when it encounters a binary/byte sequence that matches a Unicode character - but that is still in the same, current, font encoding (OT1 or T1 in the MWE above).

So what is needed in order to typeset a Euro sign - regardless if you enter it as ^^e2^^82^^ac or as €, as now we discuss when [utf8]{inputenc} is enabled, - is that by \DeclareUnicodeCharacter some macro is called, which will switch temporarily to TS1, and output glyph number 191 from there. That seems to be exactly what the textcomp package does (although I cannot figure out the numbering details):

$ grep euro `kpsewhich utf8enc.dfu`
\DeclareUnicodeCharacter{20AC}{\texteuro}

$ grep euro `kpsewhich textcomp.sty`
    \space\space 4 = 5 + \string\texteuro\MessageBreak
\DeclareOption{euro}{\DeclareEncodingSubset{TS1}{?}{4}}
\def\tc@fake@euro#1{%
\DeclareTextCommandDefault{\texteuro}
   {\CheckEncodingSubset\UseTextSymbol{TS1}\tc@fake@euro5\texteuro}

So if instead of \DeclareUnicodeCharacter{20AC}{TheEURO}, you add \usepackage{textcomp} after \usepackage[utf8]{inputenc}, you will see ^^e2^^82^^ac in the Tex source typeset as € glyph in the pdf.

To come back to the Hangul characters: if I have the characters, I can find both their Unicode indexes, and UTF-8 byte sequences:

$ echo -n 각갓갥갷 | perl utfinfo.pl 
Got 4 uchars
Char: '각' u: 44033 [0xAC01] b: 234,176,129 [0xEA,0xB0,0x81] n: HANGUL SYLLABLE GAG [Hangul Syllables]
Char: '갓' u: 44051 [0xAC13] b: 234,176,147 [0xEA,0xB0,0x93] n: HANGUL SYLLABLE GAS [Hangul Syllables]
Char: '갥' u: 44069 [0xAC25] b: 234,176,165 [0xEA,0xB0,0xA5] n: HANGUL SYLLABLE GAELG [Hangul Syllables]
Char: '갷' u: 44087 [0xAC37] b: 234,176,183 [0xEA,0xB0,0xB7] n: HANGUL SYLLABLE GAEH [Hangul Syllables]

... and using [utf8]{inputenc}, we can easily enter, say, ^^ea^^b0^^81 (the UTF-8 byte sequence in hex) in the Latex document for the '각' HANGUL SYLLABLE GAG; and even define a handler for it, using \DeclareUnicodeCharacter{AC13}{\myHangulGAGhandler} (using the Unicode index in hex) - the problem is, we'd need to know which font and in which font encoding has that glyph (and at what location), so that we could program \myHangulGAGhandler to select and typeset that glyph. And unfortunately, I know nothing of Hangul/Korean fonts (alghough, I have a CJKutf8 example in How does one type Chinese in LaTeX?, which might be useful as a starting point). In any case, using the above code to inspect font settings (also with a font table) in Latex documents, that you know are compiling well to Hangul/Korean, might be of help.

Otherwise, as far as using the Unicode index hex directly to output a character as in \unicodechar{bc73}, there seems to be no convenient macro for that; although as @UlrikeFischer noted, \inputenc already does that in the background - since it has to convert the 20AC in \DeclareUnicodeCharacter{20AC}... to a byte sequence, to which it can react later. That is mighty complex, however; you can use something like this code:

...
\usepackage[utf8]{inputenc}
\usepackage{trace}
\traceon
\DeclareUnicodeCharacter{20AC}{TheEURO}
\traceoff
...

.. to get printouts like:

\UTFviii@tmp ->\UTFviii@three@octets €
{\expandafter}
{\expandafter}

\UTFviii@three@octets #1#2#3->\csname u8:#1\string #2\string #3\endcsname 
#1<-
#2<-�
#3<-�
{\expandafter}
....

... but in a shell, that looks all wrong; if the .log file is opened in less or a hex editor, you can see:

...
\UTFviii@three@octets #1#2#3->\csname u8:#1\string #2\string #3\endcsname 
#1<-<E2>
#2<-<82>
#3<-<AC>
{\expandafter}
{\csname}
...

... that inputenc, based on this 20AC Unicode hex ID, creates a command which has the raw UTF byte sequence (0xE2,0x82,0xAC - or at least the first byte of it) in its name! (actually, I think inputenc somehow makes the byte 0xE2 an alias to the macro \UTFviii@three@octets, and in that way it "reacts" to extended UTF-8 characters). And understanding this requires knowledge not only of Latex internals, but also of actual UTF-8 encoding, which I think is partly the mechanism here - which I unfortunately don't have. Still, hope that this can be useful...