2

This question refers to how the program TeX creates tokens after reading a line of a .tex-file. It does actually not refer to writing macros or doing typesetting work which takes place in stages after tokenization.

Assume TeX is switched to verbatim catcode régime, where ^^-notation and the like doesn't work.

(Verbatim catcode régime: (space), \, {, }, $, &, #, ^, _, % and ~ all have category code 12 assigned, which is "other". No character has assigned one of the category codes 0, 1, 2, 3, 4, 6, 7, 8, 10, 14 and the only character that might be active or might have category 5 is the carriage return character, which might also have one of the categories 9, 11, 12 and 15.)

Can TeX, while such a catcode régime is in effect, produce—by reading lines from a .tex-file and tokenizing characters of that line— an explicit character token whose character code is 13?
(Character code 13 in TeX's internal character encoding scheme denotes a character which also is referred to as "carriage return character" or CR. So let's name such tokens explicit carriage-return-character-tokens.)


(Edit: This question is motivated by attempts of grasping why with LaTeX's verbatim environments and arguments of type +v relying on explicit carriage-return-character tokens as markers for places where in the corresponding .tex-input file a line ended, is safe. Seems it is safe because in verbatim mode such character tokens cannot come into being during tokenization other than due to the \endlinechar-mechanism. )

15
  • @DavidCarlisle TeX could be switched to verbatim catcode régime by a macro which does a lot of reading and tokenizing and under certain conditions, e.g., having encountered characters that form a specific phrase, switching back to normal caatcode-régime. Seems, this is the underlying idea with LaTeX's \verb and verbatim-environments that scan for an ending-phrase tokenized in verbatim catcode régime...
    – user301872
    Commented Mar 16 at 17:16
  • @DavidCarlisle If you combine a verbatim catcode régime where the carriage return character is assigned category 12 while \endlinechar is 13, characters inserted due to the endlinechar-mechanism get tokenized as explicit character tokens of category 12(other)... Commented Mar 16 at 17:20
  • @DavidCarlisle If I understand correctly, then "might" indicates that it also might not be active. Probably the questioner is willing to clarify what "might" might mean... ;-) Commented Mar 16 at 17:25
  • @DavidCarlisle "the only character that might be active or might have category 5 is the carriage return character": No character other than the carriage return character might have one of these categories. The carriage return character is the only character that might have one of these categories. It might also have one of the categories 9, 11, 12 and 15.
    – user301872
    Commented Mar 16 at 17:33
  • @Cattleya you clearly have some processing setup in mind your question would be a lot clearer if you included an example input and then asked if that could produce active ^^M as it is, I really can not even guess what this question means Commented Mar 16 at 17:40

1 Answer 1

0

(This answer does not take the possibilities of the Lua-backend of a LuaTeX-engine into account.)

Simplified speaking:

With the TeX implementations known to the author of the initial release of this answer1, the routines for reading a line from a text file (beneath other things) don't take carriage return characters occurring in a text file for something that belongs to the characters written "into" a line of text. But the routines for reading a line from a text file take carriage return characters occurring in the text file for something that denotes that subsequent stuff in the text file belongs to subsequent lines of the text file.

Thus, with the TeX implementations known to the author of the initial release of this answer, carriage return characters occurring in a text file don't make it into what, after reading a line from a text file, by TeX is processed as "the characters of the line of the text file" while characters of the text file that during reading from the text file don't make it into what is considered "the characters of the line of the text file" are not subject to treatment during tokenization.

With the TeX implementations known to the author of the initial release of this answer, in the context of reading text files, only two possibilities are known to the author of the initial release of this answer, how the circumstance of a carriage return character being subject to treatment during tokenization can come into being:

Possibility 1:
Right after reading a line of a text file, within in the stage of pre-processing the characters of that line, where the category code régime is not yet effective, TeX's \endlinechar-mechanism may have caused the appending of a carriage return character to "the characters of the line of the text file". That carriage return character is subject to treatment during tokenization and hereby is treated according to the catcode régime current at the time of treating.
(With catcode régimes where ^^-notation is not possible, this is the only possibility of how the circumstance of a carriage return character being subject to treatment during tokenization can come into being.)

Possibility 2:
With catcode régimes where ^^-notation is possible, the circumstance of a carriage return character being subject to treatment during tokenization can come into being during the stage of tokenization, i.e., after pre-processing, due to encountering the carriage return character's ^^-representation, i.e. ^^M or ^^0d, and replacing the characters forming that ^^-representation by a carriage return character and subsequently treating that carriage return character according to the catcode régime current at the time of treating.

So, with the TeX implementations known to the author of the initial release of this answer, the circumstance of a carriage return character being subject to treatment during tokenization is never due to TeX having found it in what, right after reading a line of a text file, TeX considers the characters of that line of the text file.

----
1The author of the initial release of this answer only knows web2c implementations of TeX where ⟨linefeed⟩ and ⟨carriage return⟩ and ⟨carriage return⟩⟨linefeed⟩ are treated as line ends.

You must log in to answer this question.