pandoc does not recognize Chinese characters

Question

I want to compile, with pandoc, a Markdown document containing CJK elements (Chinese, actually). It was stated there that --latex-engine=xelatex option allows pandoc to compile Unicode characters.

However, I tried
cjk.md:

Hello
你好

compiled with (in bash)

pandoc -s -o cjk.pdf --latex-engine=xelatex cjk.md

But the resulting .pdf has Hello shown only, while 你好 is not shown. Have I missed something?

(Yesterday I asked this in Stack overflow, but no one paid attention to me.... and people there are scary to me ><" )

background: I plan to ask quite many question in Math SE, and want to minimize the trouble converting them to another file convenient to compile on local. Mandarin is my native, I sometimes desire to add comments in Chinese (for my own reference only). If I cannot find a easy way to compile, on local, markdown documents embedded with LaTeX and strewn with few Chinese words, my motivation to do all these would be discouraged.

Answer 1

The conversion to PDF uses LaTex as the backend. You will need to specify a suitable mainfont with chinese glyphs. You can do this by putting the following section at the top of your .md document, e.g.

---
mainfont: Noto Sans Mono CJK TC
---

Then compile with the --latex-engine=xelatex switch and you should have your desired output. Note: run fc-list from a terminal to see which CJK fonts are installed.

Answer 2

There is no real need for modifying the Pandoc LaTeX template if you just need to render Chinese characters in your document. The following compiles fine:

---
CJKmainfont: Noto Sans CJK SC Regular
CJKoptions: AutoFakeBold
---

Hello
你好

**你好**

I used the following command to compile the above,

pandoc -t latex -o cjk.pdf --latex-engine=xelatex cjk.md

I've highlighted the required part of the intermediate LaTeX file generated as follows:

\else % if luatex or xelatex
  \ifxetex
    \usepackage{mathspec}
  \else
    \usepackage{fontspec}
  \fi
  \defaultfontfeatures{Ligatures=TeX,Scale=MatchLowercase}
    \usepackage{xeCJK}
    \setCJKmainfont[AutoFakeBold]{Noto Sans CJK SC Regular}
\fi

If you really intend to edit the Pandoc template, then it's better to make it flexible for future use. Instead of hard-coding your options, introduce variables instead. Thus,

$if(CJKmainfont)$
    \usepackage{xeCJK}
    \setCJKmainfont[$for(CJKoptions)$$CJKoptions$$sep$,$endfor$]{$CJKmainfont$}
    $if(CJKsansfont)$
      \setCJKsansfont[$for(CJKsansoptions)$$CJKsansoptions$$sep$,$endfor$]{$CJKsansfont$}
    $endif$
    $if(CJKmonofont)$
      \setCJKmonofont[$for(CJKmonooptions)$$CJKmonooptions$$sep$,$endfor$]{$CJKmonofont$}
    $endif$
$endif$

We can then use the following source as input through the template,

---
CJKmainfont: STSong
CJKoptions: BoldFont=STHeiti,ItalicFont=STKaiti

CJKsansfont: STXihei
CJKsansoptions: BoldFont=STHeiti

CJKmonofont: STFangsong
---

Hello
你好

**你好**

which would give us the intermediate LaTeX file as follows (I've just highlighted the important part):

\else % if luatex or xelatex
  \ifxetex
    \usepackage{mathspec}
  \else
    \usepackage{fontspec}
  \fi
  \defaultfontfeatures{Ligatures=TeX,Scale=MatchLowercase}
    \usepackage{xeCJK}
    \setCJKmainfont[BoldFont=STHeiti,ItalicFont=STKaiti]{STSong}
          \setCJKsansfont[BoldFont=STHeiti]{STXihei}
              \setCJKmonofont[]{STFangsong}
    \fi

This way, your template is more flexible as these options can be changed directly from the Markdown files.

Answer 3

Solved. I typed that in Stack Overflow, here. But SE dislikes me just post a link (and no more), so I now copy it here.

---

Tl;dr: it's not enough to set compiler to be xelatex instead of pdflatex; one has to include package xeCJK as well, but for where it should be, see below.

---

In my case, pandoc reads a .md, converts it to be a .tex, and call compiler to compile that to be a .pdf. Thus, in theory, what I can do normally with a tex-like compiler, can be done with pandoc as well --- it is only that I have to specify required template.

The pandoc calls its own latex template, which we can cat in the terminal by a prewritten command:

pandoc -D latex > default.latex

This essentially copies a new file default.latex to . (current directory). It is this I now modify. Append this option to pandoc when compiling:

--template=my-directory/my-template.latex

In the past I type Chinese characters (or more generally, CJK), I use a template beginning with

\documentclass[12pt]{article}
\usepackage{xeCJK}% use Latin font whenever possible
\usepackage{fontspec}% set Chinese fonts, as follows
\setCJKmainfont[BoldFont=STHeiti,ItalicFont=STKaiti]{STSong}
\setCJKsansfont[BoldFont=STHeiti]{STXihei}
\setCJKmonofont{STFangsong}
% .... whatever xeCJK commands you use

The fonts should be those your system permits; these shown above are shipped with mac.

But when I simply pasted this into the pandoc-provied template, there were many cryptic error messages

option clash for package XXXX....

This was because the pandoc-proviede template already defined xeCJK. Indeed, search these lines:

$if(CJKmainfont)$
    \usepackage{xeCJK}
    \setCJKmainfont[$for(CJKoptions)$$CJKoptions$$sep$,$endfor$]{$CJKmainfont$}
$endif$

These lines (quoted part) should be replaced by

\usepackage{xeCJK}
\setCJKmainfont[BoldFont=STHeiti,ItalicFont=STKaiti]{STSong}
% .... and so on, whatever you call from xeCJK

i.e., delete if, so that xeCJK is always executed; otherwise, xeCJK line will not be copied to the intermediate .tex file. And also delete \usepackage{fontspec}, because it is called by pandoc by default, otherwise error message occurs for packages are called twice in the intermediate .tex file.

---

acknowledgement: [1] [2] [3]

Sorry I didn't keep track of every websites from which I referenced, but none gets it all right anyway, or are outdated. Of most help is mb21, who suggested in the comment that I output .tex to debug, after which I found xeCJK was not included.

I have spent some 10+ hrs on this issue, but from now on I can happily type Chinese in a markdown file. I have wrote this down for poor posterity's sake.