What is kcatcode in pTeX and upTeX?

Question

The Japanese TeX engines pTeX and upTeX treat separately Latin and Japanese characters, in a way I don't fully understand. They introduce a notion of kcatcode of a Japanese character, which takes values 16, 17, 18 (in pTeX) or 19 (in upTeX) EDIT: also 15 as described in the answer below. What do these values correspond to? The \meaning of a character with these different \kcatcode values is the same: the following shows > kanji character 本. four times (and an error in pTeX since 19 is not allowed).

\kcatcode`本=16
\show 本
\kcatcode`本=17
\show 本
\kcatcode`本=18
\show 本
\kcatcode`本=19
\show 本

In addition, the kcatcode values 16,17,19 seem to be allowed in control sequences.

Answer 1

Basics of \kcatcode

In pTeX/upTeX, all Japanese characters are classified into some pre-defined groups [*1], and each of these groups is given a \kcatcode value. The value works as a marker which means that the character is treated as a Japanese character token. It is also referred to by pTeX/upTeX in various situations, including the input processor and the line breaking routine.

Meanings of \kcatcode value

Here are the meanings of \kcatcode values ​​in pTeX/upTeX:

• 16: Kanji characters (e.g. 漢 日 ...)
• 17: Kana (Hiragana, Katakana) characters (e.g. あ ア)
• 18: Symbols (e.g. 、 。)
• 19: Hangul characters (e.g. 아 하) -- this is only available in upTeX

In upTeX, a special \kcatcode value 15 is also allowed; it means that the character is not interpreted as a Japanese character token, and a normal \catcode value becomes effective. This feature is meant for compatibility with Western TeX, including UTF-8 handling of the inputenc package (see also "6.6.2 Detecting upTeX" in ptex-guide-en.pdf).

The characters which have kcatcode 16,17,19 are allowed in control sequences (similar to catcode 11). The characters which have kcatcode 18 are allowed in control symbols (similar to catcode 12). Regardless of all of these, the result of \meaning for any Japanese character will always look like kanji character <something>.

---

Note:

[*1] Unlike \catcode, the \kcatcode settings are effective not for individual characters, but for some pre-defined block of characters. (e.g. both "あ" and "い" are Hiragana characters, so they share the same \kcatcode value.) Also, the classification is a bit different between pTeX and upTeX (pTeX: based on JIS ku-ten codes, upTeX: based on Unicode blocks), which is hardwired in the WEB source.

---

See also: H. Kitagawa, "Distinguishing 8-bit characters and Japanese characters in (u)pTeX" (TUGboat 41:3, 2020) https://tug.org/TUGboat/Contents/contents41-3.html --- The mentioned "experimental version of (u)pTeX" is not merged yet.

See also: T. Tanaka, "upTeX --- Unicode version of pTeX with CJK extensions" (TUGboat 34:3, 2013) https://tug.org/TUGboat/Contents/contents34-3.html