3

The inputenc-package provides means for selecting an input-encoding for subsequent tex-input.

If I understand correctly, the verbatim-environment as of the LaTeX 2ε-kernel does change some catcodes so that things get tokenized as ordinary character-tokens and then does gather lines of input until it finds a line which contains the phrase \end{verbatim}.

My question is:

Is the assumption correct that you can use the verbatim-environment only with such input-encodings where the symbols \, e, n, d, {, v, r, b, a, t, i, m, } have the same code-points as in the encoding in which the LaTeX 2e-kernel itself is written?

(This is the case with all common input-encodings, thus the question is sort of a moot point. Nonetheless I'm thankful for clarification.)

2 Answers 2

1

What do you mean by "the encoding in which LaTeX 2e-kernel itself is written" ?

What do you mean by "LaTeX 2ε-kernel" in this context?

Do you mean the text-file latex.ltx from which, e.g., a format-file/.fmt-file can be dumped or a core-dump or a binary can be created/produced?

Regarding the format-file/the core-dump/the binary: During tokenization, at the pre-processing stage, everything gets converted to (La)TeX's internal character representation scheme (, which is ASCII with conventional (La)TeX-engines and which is UTF-8 with modern TeX-engines like Lua(La)TeX or Xe(La)TeX ).

Therefore the encoding in which latex.ltx was written does not matter any more when the format-file is dumped or the binary is created.

What matters is:

When the (la)tex-engine does read, pre-process and tokenize your input, this must yield that set of tokens which, according to the definitions provided in terms of tokens via the LaTeX 2ε-kernel, the (la)tex-engine does search for detecting the end of the verbatim-environment.

Be that as it may.


A dangerous-bend-paragraph in Chapter 8: The Characters You Type of Donald Ervin Knuth's TeXbook says:

TeX always uses the internal character code of Appendix C for the standard ASCII characters, regardless of what external coding scheme actually appears in the files being read. Thus, b is 98 inside of TeX even when your computer normally deals with EBCDIC or some other non-ASCII scheme; the TeX software has been set up to convert text files to internal code, and to convert back to the external code when writing text files. Device-independent (dvi) output files use TeX’s internal code. In this way, TeX is able to give identical results on all computers.

With (La)TeX the very first stage of processing input is reading that input and pre-processing and tokenizing that input and putting the resulting tokens into the token-stream.

The source whereof (La)TeX reads and pre-processes and tokenizes input can be a text file or the console where you type input "on the fly".

Converting characters from external code to internal code at the time of reading and pre-processing input right before tokenizing may imply replacing the bytes read from the input by other bytes before obtaining tokens. The routine for converting the characters that occur in text files/in the input from external code to internal code does not take place on "token-level" but it takes place on "input-character-level", i.e., in that stage where TeX reads characters from the input file for tokenizing them (and as a result of tokenizing inserting tokens into the token-stream).

There are different TeX-distributions.
E.g., in the manual of the TeX-distribution MiKTeX (MiKTeX Manual, Revision 2.9.6809, Christian Schenk, https://miktex.org/download/ctan/systems/win32/miktex/doc/2.9/miktex.pdf) you can read that with the TeX-binaries of MiKTeX the rules for converting to internal code (ASCII) and converting back to external code can be determined by means of so-called .tcx-files (TeX character translation-files).

The inputenc-package works only with 8-bit-engines where the routine for converting to internal code does not replace bytes read from the input by other bytes. In other words: The inputenc-package works only with 8-bit-engines where the external code/the external encoding contains the ASCII-code as a subset.
This is because the inputenc-package performs its own character-replacement-routines. It does not perform them on "input-character-level" but it does perform them on "token-level" by making characters active and defining them to expand to non-active characters of different character-code. That replacement-routine, which runs on token-level, does not have information about what byte-replacements beforehand did take place on "input-character-level" due to TeX's routines for converting characters from external code to internal code.

Assume you use a LaTeX 2ε-engine which by default does process latin1-encoded files (7-bit-ASCII is a subset of 8bit-latin 1). With such an engine converting from external code to internal code does not imply the need of replacing bytes of the input by other bytes.
Loosely speaking, with such an engine the character-codes of character-tokens which came into being due to tokenizing the characters in the .tex-input-file correspond to the numbers of the code-points of the corresponding input-characters in latin1-encoding.

You can't do this but—just for the sake of having fun—assume that with this engine you can load the inputenc-package so that you can "feed" to the engine a file which is encoded in an encoding whereof ASCII is not a subset, e.g., some EBCDIC variant:

The numbers of the code-points that in EBCDIC encode the character-sequence \end{verbatim} are not the same as the numbers of the code-points that in ASCII encode the character-sequence \end{verbatim}. The engine will at the time of tokenization not perform any transformation, thus it will take the EBCDIC-Bytes from the input for ASCII-Bytes and therefore will tokenize the EBCDIC-encoded input-sequence \end{verbatim} as a sequence of character-tokens whose character-codes correspond to the numbers of the code-points of these characters in EBCDIC-encoding. But the mechanism of the verbatim-environment will search for a sequence of character tokens whose character codes correspond to the numbers of the code-points of these characters in ASCII-encoding.
Therefore the verbatim-mechanism would be broken.

But this problem would not be an issue because it would be overridden by a much bigger problems:

E.g., the bytes that in ASCII encode the backslash and the name a of control-sequence-token will not be the same as the bytes that in EBCDIC encode the same. But inputenc's routine comes into action when tokenizing is over. Therefore in this scenario sequences of EBCDIC bytes that in EBCDIC-encoding denote backslashes and names of control-sequences are erroneously taken for ASCII-bytes. E.g., the EBCDIC-byte for encoding the backslash does in ASCII/latin 1 not encode the backslash, therefore tokenizing EBCDIC encoded sequences that are intended to denote control-sequence-tokens will not get tokenized as control-sequence tokens but might get tokenized as whatsoever character-token-sequences.

Besides this category-codes might be an issue as well: Inputenc does its conversions by making characters active and defining them to expand to other non-active characters. Thus you'd need to have at least all those characters active where the numbers of their code-points when taken for ASCII denote other characters than when taken for EBCDIC. Thus in this scenario the result of tokenizing an EBCDIC-encoded sequence of input-characters which shall denote a control-sequence-token might contain some active character tokens which actually should not have been tokenized as such but should have been taken for a part of the name of a control sequence token whereof the byte-sequence forming the name should have been transformed from EBCDIC to internal code=ASCII/UTF-8 but wasn't.

In the context of this problem the paper "LaTeX 2ε Encoding Interfaces—Purpose, Concepts, and Open Problems" by Frank Mittelbach, Brno, June 1995, https://www.latex-project.org/publications/1995-FMi-CsTUG-encoding-concepts.pdf, might be of interest to you: Section 7: Input Encodings: The LaTeX 2ε solution says:

  • 7-bit printable ASCII is considered to be essentially transparent
  • Input encodings are declared by name
  • Each named encoding defines the mapping for (a subset of) the character numbers between 128–255 to the internal encoding of LaTeX 2ε
  • Input encodings are declared for a whole document or for parts of it

Exceptions are the standard escape characters for LaTeX (since they are not passed through) and the non-printable ASCII characters at the moment. This means, for example, that encodings like EBCDIC (in any variant are not supported. This isn't totally surprising because to be able to support an input encoding it must at least have the characters that form TeX's command language in well-defined positions.

(The missing closing parenthesis after "in any variant" is not a quoting error but it is missing in the original. ;-) )

2

Your assumption is incorrect: TeX only looks at bytes.

When the setup for verbatim mode does \catcode`\\=12 to make the backslash into a printable character, TeX actually assigns category code 12 to byte 92. After the assignment has been performed, every time byte 92 is scanned when reading input, it is transformed into a character token of category code 12.

Input encoding are implemented by making some of the bytes active (category code 13) and assigning them suitable definitions. An input encoding might make \ into an active character, but that would be irrelevant when verbatim mode is being processed.

If you're referring to exotic encodings such as EBCDIC, the point is again irrelevant, because all bytes are translated into ASCII before being examined for tokenization.

You must log in to answer this question.