9

The Japanese TeX engines pTeX and upTeX treat separately Latin and Japanese characters, in a way I don't fully understand. They introduce a notion of kcatcode of a Japanese character, which takes values 16, 17, 18 (in pTeX) or 19 (in upTeX) EDIT: also 15 as described in the answer below. What do these values correspond to? The \meaning of a character with these different \kcatcode values is the same: the following shows > kanji character 本. four times (and an error in pTeX since 19 is not allowed).

\kcatcode`本=16
\show 本
\kcatcode`本=17
\show 本
\kcatcode`本=18
\show 本
\kcatcode`本=19
\show 本

In addition, the kcatcode values 16,17,19 seem to be allowed in control sequences.

1
  • hm texdoc ptex-guide-en implies the 16-19 kcatcodes exist but not what they mean – David Carlisle May 24 at 11:10
9

Basics of \kcatcode

In pTeX/upTeX, all Japanese characters are classified into some pre-defined groups [*1], and each of these groups is given a \kcatcode value. The value works as a marker which means that the character is treated as a Japanese character token. It is also referred to by pTeX/upTeX in various situations, including the input processor and the line breaking routine.

Meanings of \kcatcode value

Here are the meanings of \kcatcode values ​​in pTeX/upTeX:

  • 16: Kanji characters (e.g. 漢 日 ...)
  • 17: Kana (Hiragana, Katakana) characters (e.g. あ ア)
  • 18: Symbols (e.g. 、 。)
  • 19: Hangul characters (e.g. 아 하) -- this is only available in upTeX

In upTeX, a special \kcatcode value 15 is also allowed; it means that the character is not interpreted as a Japanese character token, and a normal \catcode value becomes effective. This feature is meant for compatibility with Western TeX, including UTF-8 handling of the inputenc package (see also "6.6.2 Detecting upTeX" in ptex-guide-en.pdf).

The characters which have kcatcode 16,17,19 are allowed in control sequences (similar to catcode 11). The characters which have kcatcode 18 are allowed in control symbols (similar to catcode 12). Regardless of all of these, the result of \meaning for any Japanese character will always look like kanji character <something>.


Note:

[*1] Unlike \catcode, the \kcatcode settings are effective not for individual characters, but for some pre-defined block of characters. (e.g. both "あ" and "い" are Hiragana characters, so they share the same \kcatcode value.) Also, the classification is a bit different between pTeX and upTeX (pTeX: based on JIS ku-ten codes, upTeX: based on Unicode blocks), which is hardwired in the WEB source.


See also: H. Kitagawa, "Distinguishing 8-bit characters and Japanese characters in (u)pTeX" (TUGboat 41:3, 2020) https://tug.org/TUGboat/Contents/contents41-3.html --- The mentioned "experimental version of (u)pTeX" is not merged yet.

See also: T. Tanaka, "upTeX --- Unicode version of pTeX with CJK extensions" (TUGboat 34:3, 2013) https://tug.org/TUGboat/Contents/contents34-3.html

7
  • Nice answer. It sounds like you are saying these are more like character codes than catcodes? It also sounds like a character's \kcatcode is not meant to be redefined. – Steven B. Segletes May 24 at 11:59
  • @StevenB.Segletes No \kcatcode is more like \catcode as it defines "how to interpret/handle a character". There are only few cases that \kcatcode needs to be changed (someone might change \kcatcode in order to allow Japanese symbols into control sequences, but it might break some aspects of Japanese-specific line breaking routine). – Hironobu YAMASHITA May 24 at 12:08
  • Thank you for the explanation! – Steven B. Segletes May 24 at 12:12
  • 1
    @BrunoLeFloch The behavior in \let-equal JP char token is still under discussion, sorry. (see github.com/texjporg/ptex-manual/issues/4 for detail; currently \ifcat and \ifx are inconsistent in pTeX) – Hironobu YAMASHITA May 24 at 12:47
  • 1
    Thank you @HironobuYAMASHITA, I will have to think of how to best support that in l3regex and other places in expl3. – Bruno Le Floch May 24 at 21:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.