2

The inputenc-package provides means for selecting an input-encoding for subsequent tex-input.

If I understand correctly, the verbatim-environment as of the LaTeX 2ε-kernel does change some catcodes so that things get tokenized as ordinary character-tokens and then does gather lines of input until it finds a line which contains the phrase \end{verbatim}.

My question is:

Is the assumption correct that you can use the verbatim-environment only with such input-encodings where the symbols \, e, n, d, {, v, r, b, a, t, i, m, } have the same code-points as in the encoding in which the LaTeX 2e-kernel itself is written?

(This is the case with all common input-encodings, thus the question is sort of a moot point. Nonetheless I'm thankful for clarification.)

0

What do you mean by "the encoding in which LaTeX 2e-kernel itself is written" ?

What do you mean by "LaTeX 2ε-kernel" in this context?

Do you mean the text-file latex.ltx from which, e.g., a format-file/.fmt-file can be dumped or a core-dump or a binary can be created/produced?

Regarding the format-file/the core-dump/the binary: During tokenization everything gets converted to (La)TeX's internal code which is ASCII with conventional (La)TeX-engines and which is UTF-8 with modern TeX-engines like Lua(La)TeX or Xe(La)TeX.

Therefore the encoding in which latex.ltx was written does not matter any more when the format-file is dumped or the binary is created.

What matters is:

When LaTeX 2ε does read and tokenize your input, this must yield that set of tokens which, according to the definitions provided in terms of tokens via the LaTeX 2ε-kernel, LaTeX 2ε does search for detecting the end of the verbatim-environment.

Be that as it may.


A dangerous-bend-paragraph in Chapter 8: The Characters You Type of Donald Ervin Knuth's TeXbook says:

TeX always uses the internal character code of Appendix C for the standard ASCII characters, regardless of what external coding scheme actually appears in the files being read. Thus, b is 98 inside of TeX even when your computer normally deals with EBCDIC or some other non-ASCII scheme; the TeX software has been set up to convert text files to internal code, and to convert back to the external code when writing text files. Device-independent (dvi) output files use TeX’s internal code. In this way, TeX is able to give identical results on all computers.

With (La)TeX the very first stage of processing input is reading that input and tokenizing that input and putting the resulting tokens into the token-stream.

The source whereof (La)TeX reads and tokenizes input can be a text file or the console where you type input "on the fly".

Converting characters from external code to internal code at the time of reading and tokenizing input may imply replacing the bytes read from the input by other bytes before obtaining tokens. The routine for converting the characters that occur in text files/in the input from external code to internal code does not take place on "token-level" but it takes place on "input-character-level", i.e., in that stage where TeX reads characters from the input file for tokenizing them (and as a result of tokenizing inserting tokens into the token-stream).

There are different TeX-distributions.
E.g., in the manual of the TeX-distribution MiKTeX (MiKTeX Manual, Revision 2.9.6809, Christian Schenk, https://miktex.org/download/ctan/systems/win32/miktex/doc/2.9/miktex.pdf) you can read that with the TeX-binaries of MiKTeX the rules for converting to internal code (ASCII) and converting back to external code can be determined by means of so-called .tcx-files (TeX character translation-files).

The inputenc-package works only with 8-bit-engines where the routine for converting to internal code does not replace bytes read from the input by other bytes. In other words: The inputenc-package works only with 8-bit-engines where the external code/the external encoding contains the ASCII-code as a subset.
This is because the inputenc-package performs its own character-replacement-routines. It does not perform them on "input-character-level" but it does perform them on "token-level" by making characters active and defining them to expand to non-active characters of different character-code. That replacement-routine, which runs on token-level, does not have information about what byte-replacements beforehand did take place on "input-character-level" due to TeX's routines for converting characters from external code to internal code.

Assume you use a LaTeX 2ε-engine which by default does process latin1-encoded files (7-bit-ASCII is a subset of 8bit-latin 1) so that converting from external code to internal code does not imply the need of replacing bytes of the input by other bytes.

You can't do this but—just for the sake of having fun—assume that with this machine you can load the inputenc-package so that you can feed to the LaTeX-engine a file which is encoded in an encoding whereof ASCII is not a subset, e.g. some EBCDIC variant:

The bytes that in EBCDIC encode the character-sequence \end{verbatim} are not the same bytes as those that in ASCII encode the character-sequence \end{verbatim}. But the mechanism of the verbatim-environment will search for the bytes that in ASCII encode the character-sequence \end{verbatim}. Therefore the verbatim-mechanism would be broken.

But this problem would not be an issue because it would be overridden by a much bigger problem:

E.g., the bytes that in ASCII encode the name a of control-sequence-token will not be the same as the bytes that in EBCDIC encode the same name. But inputenc's routine comes into action when tokenizing is over. Therefore in this scenario byte-sequences denoting names of control-sequences are erroneously taken for ASCII-bytes while in actual fact they are EBCDIC-Bytes.
Besides this category-codes might be an issue as well: Inputenc does its conversions by making characters active and defining them to expand to other non-active characters. Thus you'd need to have at least all those characters active where the numbers of their code-points when taken for ASCII denote other characters than when taken for EBCDIC. Thus in this scenario the result of tokenizing an EBCDIC-encoded sequence of input-characters which shall denote a control-sequence-token might contain some active character tokens which actually should not have been tokenized as such but should have been taken for a part of the name of a control sequence token whereof the byte-sequence forming the name should have been transformed from EBCDIC to internal code=ASCII/UTF-8 but wasn't.

In the context of this problem the paper "LaTeX 2ε Encoding Interfaces—Purpose, Concepts, and Open Problems" by Frank Mittelbach, Brno, June 1995, https://www.latex-project.org/publications/1995-FMi-CsTUG-encoding-concepts.pdf, might be of interest to you: Section 7: Input Encodings: The LaTeX 2ε solution says:

  • 7-bit printable ASCII is considered to be essentially transparent
  • Input encodings are declared by name
  • Each named encoding defines the mapping for (a subset of) the character numbers between 128–255 to the internal encoding of LaTeX 2ε
  • Input encodings are declared for a whole document or for parts of it

Exceptions are the standard escape characters for LaTeX (since they are not passed through) and the non-printable ASCII characters at the moment. This means, for example, that encodings like EBCDIC (in any variant are not supported. This isn't totally surprising because to be able to support an input encoding it must at least have the characters that form TeX's command language in well-defined positions.

2

Your assumption is incorrect: TeX only looks at bytes.

When the setup for verbatim mode does \catcode`\\=12 to make the backslash into a printable character, TeX actually assigns category code 12 to byte 92. After the assignment has been performed, every time byte 92 is scanned when reading input, it is transformed into a character token of category code 12.

Input encoding are implemented by making some of the bytes active (category code 13) and assigning them suitable definitions. An input encoding might make \ into an active character, but that would be irrelevant when verbatim mode is being processed.

If you're referring to exotic encodings such as EBCDIC, the point is again irrelevant, because all bytes are translated into ASCII before being examined for tokenization.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy