1

I would like to know an efficient way to use Cyrillic characters in command names (macro names) with PDFLaTeX (XeLaTeX being unavailable).

I discovered that I can define commands with Cyrillic letters in their names if I keep the source in Windows-1251 8-bit character encoding as so:

\documentclass{article}

\usepackage[T2A]{fontenc}
\usepackage[cp1251]{inputenc}
\usepackage[russian]{babel}

\catcode`\л=11\relax
\catcode`\п=11\relax
\catcode`\к=11\relax

\newcommand{\лк}{«}
\newcommand{\пк}{»}

\begin{document}
\лк Золотой век\пк\ распространения славянской письменности относится ко времени царствования в Болгарии царя Симеона Великого.
\end{document}

My first question is: how "robust" is this solution?

Also, I would like to know how to set the category code to 11 for all Cyrillic letters, without doing it one by one.

Update: I have discovered that if I use hyperref package with this solution and create sections in the document, the generated PDF document outline will have wrong characters in the place of letters for which I set the catcode 11. I would like to know what could be done about this.

Update 2: I've found that there is an "unofficial" package russlh that uses exactly this approach, but its latest update was apparently in 2003, and it does not seem to work with recent versions of hyperref.

My second question is: can I have Cyrillic commands if I use UTF-8 for the source?

Simply reencoding the example file and using utf8 instead of cp1251 did not work:

! Missing number, treated as zero.
<to be read again> 
                   \protect 
l.7 \catcode`\л
                =11\relax

There is a related question about Command names with utf-8/special characters, but I did not quite understand this part of the unswer:

With pdfTeX the \√ syntax will not work because of the way pdfTeX handles input (it is theoretically possible to achieve this, though).

7
  • 1
    Imho it is not possible. At least not if you want to use the chars also for normal words. The bytes of a char like л can't have at the same time catcode "letter" (for commands) and "active" (for the inputenc processing). Why is xelatex or lualatex not available? Jul 6, 2018 at 15:49
  • cp1251 is an 8 bit encoding, so you can do the trick with \catcode(but lose the possibility to use the characters in text); it's not possible with UTF-8.
    – egreg
    Jul 6, 2018 at 16:05
  • @UlrikeFischer, it is for arXiv.
    – Alexey
    Jul 6, 2018 at 16:09
  • @egreg, my example with cp1251 compiled fine, i didn't loose the possibility to use the characters in text. However, when i tried to add hyperref package (with or without unicode option), the letters for which i manually set catcode got misrepresented in the document's outline created by hyperref.
    – Alexey
    Jul 6, 2018 at 16:14
  • 1
    In case anybody is wondering why you would want such a thing: there are very old (and, I believe, pre-babel) Russian support packages for LaTeX that use the same input and font encoding (usually KOI8-R), set Cyrillic letters to catcode 11, and use commands like \лк (from левая кавычка “left quotation mark”) for « (in modern documents the problem of inputting this character is instead solved by having a ligature for <<). I suspect the OP is trying to wrangle one of these ancient documents. Mar 19, 2021 at 14:08

3 Answers 3

2

With multibyte encodings such as UTF-8, what you want to do is basically impossible (or can be implemented in a very fragile way).

It seems to work with 8 bit encodings such as Windows-1251, but a simple example will show what can go wrong:

\documentclass{article}

\usepackage[T2A]{fontenc}
\usepackage[cp1251]{inputenc}
\usepackage[russian]{babel}

\catcode`\ё=11\relax

\begin{document}

ё

\end{document}

(this should be saved in Windows-1251 encoding). If I run it the output is

enter image description here

The reason is simple: in the Windows-1251 (cp1251 in LaTeX lingo), the character ё is encoded as 0xB8, but T2A encoded fonts have ҷ in that position (see the table below). By setting the catcode of byte 0xB8 to 11, TeX will interpret it as “print the glyph in position 0xB8 in the current font”.

The job of inputenc is indeed to make byte 0xB8 active with expansion \char"BC (the slot where the ё character lives in T2A encoded fonts).

enter image description here

5
  • 1
    Virtual fonts could perform the mapping from input encoding to output encoding slots. But they need to be created for all used fonts. If the input encoding is UTF-8 (multi-byte), then the ligature mechanism can help. Jul 6, 2018 at 18:06
  • 1
    @HeikoOberdiek With obvious problems for the hyphenation patterns.
    – egreg
    Jul 6, 2018 at 18:10
  • An adapted version of the hyphenation patterns could also be created, but the way remains cumbersome, not very portable, and therefore not recommended for 8-bit TeX engines. Jul 6, 2018 at 18:15
  • Thanks for the explanation. It seems that without using ё character in commands, a working solution could be to change the catcode of all characters except ё. Unfortunately, this still leaves me with the problem of misencoded contents outline produced by hyperref.
    – Alexey
    Jul 6, 2018 at 18:45
  • @Alexey The bookmark code has to map the characters to UTF16-BE (option unicode or pdfencoding=auto is required, because the other bookmark encoding of the PDF format is 8-bit and does not contain Cyrillic letters). A special font encoding is used to map LICRs (LaTeX internal character representation) to the bookmark. The characters outside ASCII are active. Thus, they can be redefined as macros to LICRs. Since letters with category codes 11 or 12 cannot be redefined at macro level, yu have to use macros \CYRa, \CYRb, ... to get correct bookmarks (see also \texorpdfstring). Jul 7, 2018 at 6:15
1

As an alternative I would suggest to define one command (with an ascii name) which takes the cyrillic as argument and builds a command name with \detokenize:

\documentclass{article}
\usepackage[T2A]{fontenc}
\usepackage[russian]{babel}


\newcommand\cyrcmd[1]{\csname cyrcmd\detokenize{#1}\endcsname}
\makeatletter
\@namedef{cyrcmd\detokenize{лк}}{«}
\@namedef{cyrcmd\detokenize{пк}}{»}
%% more commands
\makeatother
\usepackage{hyperref}
\begin{document}
\tableofcontents
\section{\cyrcmd{лк}Золотой век\cyrcmd{пк}}
\cyrcmd{лк}Золотой век\cyrcmd{пк} распространения славянской письменности относится ко времени царствования в Болгарии царя Симеона Великого.
\end{document}

enter image description here

0

[This is essentially an extended comment because while I have once researched the options listed here, I haven’t actually implemented any of them.]

I believe you can’t do this with inputenc and a conventional TeX engine.

Latin letters are special for TeX because they essentially have to be encoded in ASCII for TeX to work (there’s a segment in the TeX source that literally translates ASCII symbols from the host encoding to ASCII after reading them in case TeX is to run on an EBCDIC system or something like that, because TeX primitives that expect keywords like pt or plus or to compare them with literal ASCII codes, categories be damned), and all TeX fonts place Latin letters at glyph numbers equal to their ASCII codepoints. This works because TeX treats a “letter” (category code 11) character with internal TeX codepoint n (on a modern ASCII-compatible system, this usually means host codepoint n) encountered within a paragraph as an instruction to output glyph number n from the current text font. This means that the internal TeX codepoints (the “input encoding”) and glyph numbers in the current font (the “font encoding”) must match. This also means that there can never be more than 28 = 256 characters in a single TeX font.

The usual LaTeX way of solving this is a bit of a hacky workaround. First, every non-ASCII codepoint is declared to be “active” (category code 13), that is, when TeX sees character n with n > 128, it’s instructed to treat it as a single-character macro (the same way it produces a non-breakable space when it encounters ~: there’s a pair of lines in latex.ltx that says \catcode`\~=13 \def~{\nobreakspace{}}); this is done by \inputencoding and \@inpenc@loop in inputenc.sty. Second, when an input encoding is set (when inputenc is being loaded), those macros are defined to expand to other macros with well-known names by calling \DeclareInputText from inputenc.sty (e. g. in a Windows-1251 document, character 168, CYRILLIC CAPITAL LETTER IO, is defined to expand to \CYRYO by cp1251.def). (The expansion is actually \IeC{\CYRA}, but that is not important here.) These well-known macro names constitute the LaTeX internal character representation (LICR). Third, when a font encoding is set (when babel is being loaded or is instructed to switch languages, or when you change \fontencoding explicitly), the LICR commands for characters that are present in that encoding are defined to emit the corresponding glyphs in the current font by calling \DeclareTextSymbol from latex.ltx (e. g. in a T2A font, \CYRYO is defined to emit character 156 by t2aenc.def). (The actual setup is more complicated, but that is again not important here, consult the ltoutenc.dtx section in source2e for details.)

Multibyte encodings such as UTF-8 are even more complicated because the leading bytes of the encoding are defined to consume the requisite number of tokens from the input stream as undelimited macro arguments, then look up and emit the corresponding LICR code. That’s why you get so many fun errors when you forget to delimit your single UTF-8-encoded Cyrillic letter passed as an undelimited argument to a macro (e. g. *\textit ы fails while \textit g works): the Cyrillic letter is not a single TeX token, and furthermore the first token that was passed to the command ate some arbitrary part of the command’s code in an attempt to fetch the trailing bytes.

To be able to use multiple Cyrillic letters in TeX commands, they need to get translated to internal TeX codepoints that declared to be “letters” (category code 11); that’s how the TeX scanner handles the backslash character (or really any “escape”, category code 0, character). So I see several approaches to accomplish what you want, and all of them are rather flaky:

  • Use a single-byte encoding like Windows-1251, IBM 866 or (historically most popular) KOI8-R; avoid inputenc and instead define them to be letters; use a font that’s already in the requisite encoding (none on a modern LaTeX installation AFAIK) or assemble an incompatible font’s characters in the necessary order by defining a virtual font and a font encoding;
  • Use a single-byte encoding or UTF-8; avoid inputenc and instead use the (apparently obscure) encTeX extension by regenerating the LaTeX format while passing the -enc option to initex; explain your desired input encoding to encTeX, which will tweak the TeX kernel’s input translation mechanism accordingly, and define the result of that translation to be letters; then hack the LaTeX kernel to recognize the result of that translation;
  • Use UTF-8 (exclusively); avoid inputenc and instead use a TeX engine that is hardwired to use UTF-8 such as XeTeX or LuaTeX; hacking the LaTeX kernel will again be probably required, but I don’t know much of anything about these engines.

If that sounds hazy, that’s because it is. I don’t know how to do this in detail, and unless you have literal hundreds of pages of old TeX documents or need to stay compatible with truly ancient TeX installations, porting your documents (possibly programmatically using a good-enough parser like plasTeX or even a good ol’ pile of regular expressions) is probably easier.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.