3

How can I get the UTF-16 code of a Chinese character in Latex? For example, if I want to print the Unicode of `你' in Latex, how can this be done by asking Latex to search the code for me?

  • Welcome to TeX.SX! What compiler are you using? With xelatex you should be able to just input the Unicode character and have it printed with a proper font loaded. Or do you actually want to print the codepoint number in your document? – siracusa Mar 18 at 3:06
  • @siracusa, I think the OP wants to get LaTeX to convert 你 to it's UTF-16 representation: 0x4F60 – David Purton Mar 18 at 9:27
  • Since the code point in this case is less than U+FFFF you can just convert from character to integer to hex which is easy enough. If you want to make a generic function which can deal with higher code points you need to be able to manipulate high and low bits which I don't know how to do in LaTeX… – David Purton Mar 18 at 10:18
2

Here's an implementation that allows to print the Unicode code point, the UTF8 representation and the UTF16 representation.

\documentclass{article}

\usepackage{xparse}
\usepackage{fontspec}

\setmainfont{Heiti SC}

\ExplSyntaxOn

\NewDocumentCommand{\codepoint}{m}
 {
  U+\xiang_codepoint:n { #1 }
 }
\NewDocumentCommand{\codepointutfviii}{m}
 {
  UTF8:\nobreakspace
  \xiang_codepoint_utfviii:f { \char_codepoint_to_bytes:n { `#1 } }
 }
\NewDocumentCommand{\codepointutfxvi}{m}
 {
  UTF16:\nobreakspace
  \xiang_codepoint_utfxvi:n { #1 }
 }

\NewDocumentCommand{\chardetails}{m}
 {
  #1~\texttt
   {
    \codepoint{#1},~
    \codepointutfviii{#1},~
    \codepointutfxvi{#1}
   }
 }


\cs_new:Nn \xiang_codepoint:n
 {
  \int_compare:nT { `#1 < 4096 } { 0 }
  \int_compare:nT { `#1 < 256 } { 0 }
  \int_to_Hex:n { `#1 }
 }

\cs_new:Nn \xiang_codepoint_utfviii:n
 {
  \tl_map_function:nN { #1 } \__xiang_codepoint_utfviii_aux:n
 }
\cs_generate_variant:Nn \xiang_codepoint_utfviii:n { f }

\cs_new:Nn \__xiang_codepoint_utfviii_aux:n
 {
  \tl_if_empty:nF { #1 }
   {
    <\int_to_Hex:n { #1 }>
   }
 }

\cs_new:Nn \xiang_codepoint_utfxvi:n
 {
  \int_compare:nTF { `#1 < 65536 }
   {
    <\xiang_codepoint:n { #1 }>
   }
   {
    \__xiang_codepoint_utfxvi:n { #1 }
   }
 }

\cs_new:Nn \__xiang_codepoint_utfxvi:n
 {
  <
  \int_to_Hex:n
   {
    \int_div_truncate:nn { `#1 - "10000 } { "400 } + "D800
   }
  >
  <
  \int_to_Hex:n
   {
    \int_mod:nn { `#1 - "10000 } { "400 } + "DC00
   }
  >
 }

\ExplSyntaxOff

\begin{document}

\chardetails{a}

\chardetails{à}

\chardetails{。}

\chardetails{豈}

\chardetails{你}

\chardetails{𠀡}

\end{document}

enter image description here

  • Much better (I do theology, not maths or comp sci, so no shame on me!). Can you improve my \__xiang_utf_viii_char_to_codepoint:n function so it will work with pdflatex? Also good to see how to use \char_codepoint_to_bytes:n, although can this work with pdflatex? @Xiang, you should definitely use this code and not mine. – David Purton Mar 20 at 3:18
1

Here's an implementation in expl3. See comments for how it works.

Update to work with all TeX engines. (The characters themselves are only printed with the Unicode engines.)

I have no idea if this is the best way of doing this…

\documentclass{article}

\usepackage{xparse}

\ExplSyntaxOn

\int_new:N \l__xiang_high_int
\int_new:N \l__xiang_low_int
\int_new:N \g__xiang_codepoint_int
\int_new:N \g__xiang_utfviii_byte_int
\intarray_new:Nn \g__xiang_utfviii_bytes_intarray { 4 }
\tl_new:N \l__xiang_bin_tl
\tl_new:N \l__xiang_high_tl
\tl_new:N \l__xiang_low_tl
\tl_new:N \l__xiang_byte_i_str
\tl_new:N \l__xiang_byte_ii_str
\tl_new:N \l__xiang_byte_iii_str
\tl_new:N \l__xiang_byte_iv_str

\cs_new:Nn \__xiang_utf_viii_char_to_codepoint:n
  {
    \int_set:Nn \g__xiang_codepoint_int { 0 }
    \int_set:Nn \g__xiang_utfviii_byte_int { 0 }
    \exp_args:NNx \str_set:Nn \l_tmpa_str { \tl_to_str:n { #1 } }
    % add each byte from UTF-8 character to array
    \str_map_inline:Nn \l_tmpa_str
      {
        \int_incr:N \g__xiang_utfviii_byte_int
        \intarray_gset:Nnn \g__xiang_utfviii_bytes_intarray
          { \g__xiang_utfviii_byte_int } { `##1 }
      }
    % calculate code point from UTF-8 bytes
    \int_case:nn { \g__xiang_utfviii_byte_int }
      {
        { 1 }
        {
          \int_set:Nn \g__xiang_codepoint_int
            { \intarray_item:Nn \g__xiang_utfviii_bytes_intarray { 1 } }
        }
        { 2 }
        {
          \int_set:Nn \g__xiang_codepoint_int
            {
              ( \intarray_item:Nn \g__xiang_utfviii_bytes_intarray { 1 } - "C0 )
              * "40 +
              ( \intarray_item:Nn \g__xiang_utfviii_bytes_intarray { 2 } - "80 )
            }
        }
        { 3 }
        {
          \int_set:Nn \g__xiang_codepoint_int
            {
              ( \intarray_item:Nn \g__xiang_utfviii_bytes_intarray { 1 } - "E0 )
              * "1000 +
              ( \intarray_item:Nn \g__xiang_utfviii_bytes_intarray { 2 } - "80 )
              * "40 +
              ( \intarray_item:Nn \g__xiang_utfviii_bytes_intarray { 3 } - "80 )
            }
        }
        { 4 }
        {
          \int_set:Nn \g__xiang_codepoint_int
            {
              ( \intarray_item:Nn \g__xiang_utfviii_bytes_intarray { 1 } - "F0 )
              * "40000 +
              ( \intarray_item:Nn \g__xiang_utfviii_bytes_intarray { 2 } - "80 )
              * "1000 +
              ( \intarray_item:Nn \g__xiang_utfviii_bytes_intarray { 3 } - "80 )
              * "40 +
              ( \intarray_item:Nn \g__xiang_utfviii_bytes_intarray { 4 } - "80 )
            }
        }
      }
  }

\cs_new:Nn \__xiang_codepoint_to_utf_xvi:n
  {
    \int_compare:nTF
      { #1 < 55296 }
      {
        % code point between U+0000 and U+D7FF
        % convert to hex
        \exp_args:NNx \tl_set:Nn \l_tmpa_tl { \int_to_Hex:n {#1} }
        % pad to 4 places
        \int_while_do:nn
          { \tl_count:N \l_tmpa_tl < 4 }
          { \tl_put_left:Nn \l_tmpa_tl { 0 } }
        % output UTF-16 in hex
        0x \tl_use:N \l_tmpa_tl
      }
      {
        \int_compare:nTF
          { 57344 <= #1 <= 65535 }
          {
            % code point between U+E000 and U+FFFF
            % output UTF-16 in hex
            0x \int_to_Hex:n {#1}
          }
          {
            \int_compare:nTF
              { #1 > 65535 }
              {
                % code point between U+010000 and U+10FFFF
                % subtract 0x10000
                \int_set:Nn \l_tmpb_int { #1 - 65536 }
                % convert to binary
                \exp_args:NNx \tl_set:Nn \l__xiang_bin_tl
                  { 
                    \int_to_bin:n { \l_tmpb_int }
                  }
                % pad to 20 bits
                \int_while_do:nn
                  { \tl_count:N \l__xiang_bin_tl < 20 }
                  {
                    \tl_put_left:Nn \l__xiang_bin_tl { 0 }
                  }
                % get high 10 bits
                \tl_set:Nn \l__xiang_high_tl
                  {
                    \tl_range:Nnn \l__xiang_bin_tl { 1 } { 10 }
                  }
                % get low 10 bits
                \tl_set:Nn \l__xiang_low_tl
                  {
                    \tl_range:Nnn \l__xiang_bin_tl { 11 } { 20 }
                  }
                % convert high 10 bits back to integer and add 0xD800
                \int_set:Nn \l__xiang_high_int
                  {
                    55296 + \exp_args:Ne \int_from_bin:n { \l__xiang_high_tl }
                  }
                % convert low 10 bits back to integer and add 0xDC00
                \int_set:Nn \l__xiang_low_int
                  {
                    56320 + \exp_args:Ne \int_from_bin:n { \l__xiang_low_tl }
                  }
                % output UTF-16 in hex
                0x \int_to_Hex:n { \l__xiang_high_int }
                \space
                0x \int_to_Hex:n { \l__xiang_low_int }
              }
              {
                % code point between U+D800 and U+DFFF
                Invalid~code~point.
              }
          }
      }
  }

\NewDocumentCommand \chardetails { m }
  {
    \sys_if_engine_pdftex:TF
      {
        \__xiang_utf_viii_char_to_codepoint:n { #1 }
      }
      {
        { \Large #1 \par }
        \int_set:Nn \g__xiang_codepoint_int { `#1 }
      }
    Code~point:~ \int_use:N \g__xiang_codepoint_int \par
    UTF-16:~ \__xiang_codepoint_to_utf_xvi:n { \g__xiang_codepoint_int }
  }

\sys_if_engine_pdftex:F
  {
    \usepackage{fontspec}
    \setmainfont{Noto~Sans~CJK~SC}
  }

\ExplSyntaxOff

\begin{document}
\chardetails{a}

\chardetails{à}

\chardetails{。}

\chardetails{豈}

\chardetails{你}

\chardetails{𠀡}
\end{document}

xelatex and lualatex output

xelatex and lualatex output

pdflatex output

pdflatex output

  • Thanks David. It works! I learned that the internal code of a character can be obtained by the operator , such as 你. As a follow-up question, is it possible to make it work under pdfLatex? Calling the command \chardetails{你} under pdfLatex provides 0x00E4, but the correct UTF code should be 0x4F60. – Xiang Mar 19 at 3:08
  • @Xiang, somehow I knew you were going to ask aout pdflatex. The \__xiang_codepoint_to_utf_sixteen:n macro will work with any engine. So if there is a way to get the Unicode code point in pdflatex then it can be done. It might help if you can modify your question to show an example with your exact use case. Printing arbitrary Unicode characters in pdflatex is not worth the pain. But it might be possible to get the code point from a character and then print the UTF-16 encoding. – David Purton Mar 19 at 3:56
  • @Xiang, it is possible to get the UTF-8 encoding of any character in pdflatex and from there convert to the code point and then convert to UTF-16. So apart from actually outputting the character, I can do this. I'll have a go later tonight. – David Purton Mar 19 at 5:37
  • @Xiang, I've updated to support pdflatex. This code just doesn't print the characters in pdflatex. – David Purton Mar 19 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.