Please forgive me if this question is too basic for this forum.
I came into touch with TeX three weeks ago.

I learned that there are different TeX engines.
E.g., there are TeX, eTeX, pdfTeX, pdfeTeX, LuaTeX, XeTeX.

If I got it right, TeX, eTeX and pdfTeX deal with 8bit encodings and therefore with these engines the range of possible character codes (numerical values for primitives like \endlinechar, \newlinechar, \char, \lccode, \uccode, \catcode etc) is 0-255.

If I got it right, LuaTeX and XeTeX deal with utf8-encoding.

What are the ranges of possible character codes with these engines?

Is there a method for (expandably and) reliably detecting what engine is in use und thus which range of character codes is available?

I ask this in order to solve the following problem:

I use eTeX' \scantokens in order to get things re-tokenized.

According to the eTeX manual \scantokens is rather similar to:

\input file

This implies the insertion of the character denoted by \endlinechar at the end of each "line" that is processed by the \input-part of \scantokens. I would like to avoid or work around that endlinechar-insertion with the very last "line" of the "input".

By now I try with something like:

\scantokens{...\begingroup\ifnum\the\catcode\endlinechar>0 \ifnum\the\catcode\endlinechar>255\else\catcode\endlinechar=14 \fi\fi}\endgroup

With this code it is assumed that the range of possible character codes is 0-255 and that range is "hardcoded".

I'd like to find a method for specifying that range depending on the engine in use and depending on the mode which the engine is run in.


The input to luatex and xetex has to be well formed utf-8 encoded Unicode string so the characters will be in the range 0 to hex 10FFFF (decimal 1114111) although luatex allows higher codes than that to be constructed from lua and used for special purposes.

The normal tests for luatex and xetex are to detect that engine-specific commands such as \directlua and \XeTeXuseglyphmetrics are defined respectively

You can use \endlinechar=-1 to suppress endlinechar but I'm not sure that will always do what you want with \scantokens (it depends exactly what you want:-) (note that luatex has a primitive \scantextokens that does not add the final \endlinechar)

With luatex even though characters are in the full Unicode range, \endlinechar is at most 127

  • @UlrichDiez ah yes thanks I'll delete that suggestion – David Carlisle Dec 3 '16 at 15:12
  • @UlrichDiez yes that's why OP set catcode instead – David Carlisle Dec 3 '16 at 15:55

Seems you rely on \scantokens re-tokenizing things under normal catcode régime.

In this case I suggest inserting something that will be recognized as comment character when \scantokens performs the \input-part/retokenization-part:
\scantokens{...\begingroup\catcode`\X=14 X}\endgroup

A\scantokens{B\begingroup\catcode`\X=14 X}\endgroup X


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.