Cyrillic letters in command names with PDFLaTeX

Question

I would like to know an efficient way to use Cyrillic characters in command names (macro names) with PDFLaTeX (XeLaTeX being unavailable).

I discovered that I can define commands with Cyrillic letters in their names if I keep the source in Windows-1251 8-bit character encoding as so:

\documentclass{article}

\usepackage[T2A]{fontenc}
\usepackage[cp1251]{inputenc}
\usepackage[russian]{babel}

\catcode`\л=11\relax
\catcode`\п=11\relax
\catcode`\к=11\relax

\newcommand{\лк}{«}
\newcommand{\пк}{»}

\begin{document}
\лк Золотой век\пк\ распространения славянской письменности относится ко времени царствования в Болгарии царя Симеона Великого.
\end{document}

My first question is: how "robust" is this solution?

Also, I would like to know how to set the category code to 11 for all Cyrillic letters, without doing it one by one.

Update: I have discovered that if I use hyperref package with this solution and create sections in the document, the generated PDF document outline will have wrong characters in the place of letters for which I set the catcode 11. I would like to know what could be done about this.

Update 2: I've found that there is an "unofficial" package russlh that uses exactly this approach, but its latest update was apparently in 2003, and it does not seem to work with recent versions of hyperref.

My second question is: can I have Cyrillic commands if I use UTF-8 for the source?

Simply reencoding the example file and using utf8 instead of cp1251 did not work:

! Missing number, treated as zero.
<to be read again> 
                   \protect 
l.7 \catcode`\л
                =11\relax

There is a related question about Command names with utf-8/special characters, but I did not quite understand this part of the unswer:

With pdfTeX the \√ syntax will not work because of the way pdfTeX handles input (it is theoretically possible to achieve this, though).

Answer 1

With multibyte encodings such as UTF-8, what you want to do is basically impossible (or can be implemented in a very fragile way).

It seems to work with 8 bit encodings such as Windows-1251, but a simple example will show what can go wrong:

\documentclass{article}

\usepackage[T2A]{fontenc}
\usepackage[cp1251]{inputenc}
\usepackage[russian]{babel}

\catcode`\ё=11\relax

\begin{document}

ё

\end{document}

(this should be saved in Windows-1251 encoding). If I run it the output is

The reason is simple: in the Windows-1251 (cp1251 in LaTeX lingo), the character ё is encoded as 0xB8, but T2A encoded fonts have ҷ in that position (see the table below). By setting the catcode of byte 0xB8 to 11, TeX will interpret it as “print the glyph in position 0xB8 in the current font”.

The job of inputenc is indeed to make byte 0xB8 active with expansion \char"BC (the slot where the ё character lives in T2A encoded fonts).

Answer 2

As an alternative I would suggest to define one command (with an ascii name) which takes the cyrillic as argument and builds a command name with \detokenize:

\documentclass{article}
\usepackage[T2A]{fontenc}
\usepackage[russian]{babel}


\newcommand\cyrcmd[1]{\csname cyrcmd\detokenize{#1}\endcsname}
\makeatletter
\@namedef{cyrcmd\detokenize{лк}}{«}
\@namedef{cyrcmd\detokenize{пк}}{»}
%% more commands
\makeatother
\usepackage{hyperref}
\begin{document}
\tableofcontents
\section{\cyrcmd{лк}Золотой век\cyrcmd{пк}}
\cyrcmd{лк}Золотой век\cyrcmd{пк} распространения славянской письменности относится ко времени царствования в Болгарии царя Симеона Великого.
\end{document}

Answer 3

[This is essentially an extended comment because while I have once researched the options listed here, I haven’t actually implemented any of them.]

I believe you can’t do this with inputenc and a conventional TeX engine.

Latin letters are special for TeX because they essentially have to be encoded in ASCII for TeX to work (there’s a segment in the TeX source that literally translates ASCII symbols from the host encoding to ASCII after reading them in case TeX is to run on an EBCDIC system or something like that, because TeX primitives that expect keywords like pt or plus or to compare them with literal ASCII codes, categories be damned), and all TeX fonts place Latin letters at glyph numbers equal to their ASCII codepoints. This works because TeX treats a “letter” (category code 11) character with internal TeX codepoint n (on a modern ASCII-compatible system, this usually means host codepoint n) encountered within a paragraph as an instruction to output glyph number n from the current text font. This means that the internal TeX codepoints (the “input encoding”) and glyph numbers in the current font (the “font encoding”) must match. This also means that there can never be more than 2⁸ = 256 characters in a single TeX font.

The usual LaTeX way of solving this is a bit of a hacky workaround. First, every non-ASCII codepoint is declared to be “active” (category code 13), that is, when TeX sees character n with n > 128, it’s instructed to treat it as a single-character macro (the same way it produces a non-breakable space when it encounters ~: there’s a pair of lines in latex.ltx that says \catcode`\~=13 \def~{\nobreakspace{}}); this is done by \inputencoding and \@inpenc@loop in inputenc.sty. Second, when an input encoding is set (when inputenc is being loaded), those macros are defined to expand to other macros with well-known names by calling \DeclareInputText from inputenc.sty (e. g. in a Windows-1251 document, character 168, CYRILLIC CAPITAL LETTER IO, is defined to expand to \CYRYO by cp1251.def). (The expansion is actually \IeC{\CYRA}, but that is not important here.) These well-known macro names constitute the LaTeX internal character representation (LICR). Third, when a font encoding is set (when babel is being loaded or is instructed to switch languages, or when you change \fontencoding explicitly), the LICR commands for characters that are present in that encoding are defined to emit the corresponding glyphs in the current font by calling \DeclareTextSymbol from latex.ltx (e. g. in a T2A font, \CYRYO is defined to emit character 156 by t2aenc.def). (The actual setup is more complicated, but that is again not important here, consult the ltoutenc.dtx section in source2e for details.)

Multibyte encodings such as UTF-8 are even more complicated because the leading bytes of the encoding are defined to consume the requisite number of tokens from the input stream as undelimited macro arguments, then look up and emit the corresponding LICR code. That’s why you get so many fun errors when you forget to delimit your single UTF-8-encoded Cyrillic letter passed as an undelimited argument to a macro (e. g. *\textit ы fails while \textit g works): the Cyrillic letter is not a single TeX token, and furthermore the first token that was passed to the command ate some arbitrary part of the command’s code in an attempt to fetch the trailing bytes.

To be able to use multiple Cyrillic letters in TeX commands, they need to get translated to internal TeX codepoints that declared to be “letters” (category code 11); that’s how the TeX scanner handles the backslash character (or really any “escape”, category code 0, character). So I see several approaches to accomplish what you want, and all of them are rather flaky:

• Use a single-byte encoding like Windows-1251, IBM 866 or (historically most popular) KOI8-R; avoid inputenc and instead define them to be letters; use a font that’s already in the requisite encoding (none on a modern LaTeX installation AFAIK) or assemble an incompatible font’s characters in the necessary order by defining a virtual font and a font encoding;
• Use a single-byte encoding or UTF-8; avoid inputenc and instead use the (apparently obscure) encTeX extension by regenerating the LaTeX format while passing the -enc option to initex; explain your desired input encoding to encTeX, which will tweak the TeX kernel’s input translation mechanism accordingly, and define the result of that translation to be letters; then hack the LaTeX kernel to recognize the result of that translation;
• Use UTF-8 (exclusively); avoid inputenc and instead use a TeX engine that is hardwired to use UTF-8 such as XeTeX or LuaTeX; hacking the LaTeX kernel will again be probably required, but I don’t know much of anything about these engines.

If that sounds hazy, that’s because it is. I don’t know how to do this in detail, and unless you have literal hundreds of pages of old TeX documents or need to stay compatible with truly ancient TeX installations, porting your documents (possibly programmatically using a good-enough parser like plasTeX or even a good ol’ pile of regular expressions) is probably easier.