# How to detect the range of possible character codes?

Please forgive me if this question is too basic for this forum.
I came into touch with TeX three weeks ago.

I learned that there are different TeX engines.
E.g., there are TeX, eTeX, pdfTeX, pdfeTeX, LuaTeX, XeTeX.

If I got it right, TeX, eTeX and pdfTeX deal with 8bit encodings and therefore with these engines the range of possible character codes (numerical values for primitives like `\endlinechar`, `\newlinechar`, `\char`, `\lccode`, `\uccode`, `\catcode` etc) is 0-255.

If I got it right, LuaTeX and XeTeX deal with utf8-encoding.

What are the ranges of possible character codes with these engines?

Is there a method for (expandably and) reliably detecting what engine is in use und thus which range of character codes is available?

I ask this in order to solve the following problem:

I use eTeX' `\scantokens` in order to get things re-tokenized.

According to the eTeX manual `\scantokens` is rather similar to:

``````\toks0={...}
\immediate\openout0=file
\immediate\write0{\the\toks0}
\immediate\closeout0
\input file
``````

This implies the insertion of the character denoted by `\endlinechar` at the end of each "line" that is processed by the `\input`-part of `\scantokens`. I would like to avoid or work around that endlinechar-insertion with the very last "line" of the "input".

By now I try with something like:

``````\scantokens{...\begingroup\ifnum\the\catcode\endlinechar>0 \ifnum\the\catcode\endlinechar>255\else\catcode\endlinechar=14 \fi\fi}\endgroup
``````

With this code it is assumed that the range of possible character codes is 0-255 and that range is "hardcoded".

I'd like to find a method for specifying that range depending on the engine in use and depending on the mode which the engine is run in.

The input to luatex and xetex has to be well formed utf-8 encoded Unicode string so the characters will be in the range 0 to hex 10FFFF (decimal 1114111) although luatex allows higher codes than that to be constructed from lua and used for special purposes.

The normal tests for luatex and xetex are to detect that engine-specific commands such as `\directlua` and `\XeTeXuseglyphmetrics` are defined respectively

You can use `\endlinechar=-1` to suppress endlinechar but I'm not sure that will always do what you want with `\scantokens` (it depends exactly what you want:-) (note that luatex has a primitive `\scantextokens` that does not add the final `\endlinechar`)

With luatex even though characters are in the full Unicode range, `\endlinechar` is at most 127

• @UlrichDiez ah yes thanks I'll delete that suggestion – David Carlisle Dec 3 '16 at 15:12
• @UlrichDiez yes that's why OP set catcode instead – David Carlisle Dec 3 '16 at 15:55

Seems you rely on `\scantokens` re-tokenizing things under normal catcode régime.

In this case I suggest inserting something that will be recognized as comment character when `\scantokens` performs the `\input`-part/retokenization-part:
`\scantokens{...\begingroup\catcode`\X=14 X}\endgroup`

``````A\scantokens{B\begingroup\catcode`\X=14 X}\endgroup X

\bye
``````