Automatically fixing catcode output

Question

Using the following macro to redefine catcodes for lua blocks to make programming lua inside tex files more natural has a fault:

\def\lua{%
  \bgroup
  \catcode`\\=12
  \catcode`\{=12
  \catcode`\}=12
  \catcode`\^^M=12
  \catcode`\#=12
  \catcode`\~=12
  \catcode`\%=12
  \catcode`\_=12
  \catcode`\@=12
  \doluacode
}

\bgroup
\catcode`\|=0
\catcode`\^^M=12 %
\catcode`\\=12 %
|long|gdef|doluacode#1^^M#2\endlua{|directlua{#2}|egroup}%
|egroup

Because it redefines the codes it also redefines the codes from tex.print output. The following code will fail

\lua
    tex.print("\\mymacro")
\endlua

because \ is interpreted with a catcode of 12.

Is there any way to get tex.print to revert back to using the original tex catcodes so printing tex stuff will work WITHOUT having to redefine each character by hand(which obviously doesn't work when you are generating tex code programmatically in lua).

That is, I would like to simply run the above code with LITTLE modification and have it work correctly and not have to do stuff like tex.print(0, "\") tex.print("mymacro") which ,AGAIN, does not work for programmatically generated tex.

The only way to do this that I have of is to create a new print function that scans the string and converts the cat codes but hopefully there is a better way?

Answer 1

ConTeXt provides \startluacode and \stopluacode macros that provide the same functionality as your \lua ... \endlua macros. The main difference between the macros is that the ConTeXt macros switch to normal catcodes before calling \direclua. The code below is copied from luat-ini.mkiv

%D A few more goodies:

\unexpanded\def\startlua    {\luat_start_lua     } \let\stoplua    \relax % tex catcodes
\unexpanded\def\startluacode{\luat_start_lua_code} \let\stopluacode\relax % lua catcodes

% It might makes sense to have a \type {\directelua} so that we can avoid
% the \type {\normalexpanded} around \type {\directlua}. Something to discuss
% in the team.

\def\luat_start_lua
  {\begingroup
   \obeylualines
   \luat_start_lua_indeed}

\def\luat_start_lua_indeed#1\stoplua
  {\normalexpanded{\endgroup\noexpand\directlua\zerocount{#1}}}

\def\luat_start_lua_code
  {\begingroup
   \obeylualines
   \obeyluatokens
   \luat_start_lua_code_indeed}

\def\luat_start_lua_code_indeed#1\stopluacode
  {\normalexpanded{\endgroup\noexpand\directlua\zerocount{#1}}}

where \obeylualines is equal to \relax (a left over from previous definition?) and \obeyluatokens is defined as

\appendtoks
   \let\\\lualetterbackslash
   \let\|\lualetterbar       \let\-\lualetterdash
   \let\(\lualetterlparent   \let\)\lualetterrparent
   \let\{\lualetterlbrace    \let\}\lualetterrbrace
   \let\'\lualettersquote    \let\"\lualetterdquote
   \let\n\lualettern         \let\r\lualetterr
   \let\f\lualetterf         \let\t\lualettert
   \let\a\lualettera         \let\b\lualetterb
   \let\v\lualetterv         \let\s\lualetters
   \let\1\lualetterone       \let\2\lualettertwo
   \let\3\lualetterthree     \let\4\lualetterfour
   \let\5\lualetterfive      \let\6\lualettersix
   \let\7\lualetterseven     \let\8\lualettereight
   \let\9\lualetternine      \let\0\lualetterzero
\to \everyluacode

\def\obeyluatokens
  {\setcatcodetable\luacatcodes
   \the\everyluacode}

Answer 2

See my answer on Which Lua environment should I use with LuaTeX (LuaLaTeX)?. The quintessence is: don't use \directlua for writing Lua code. Why? Because you get exactly the problem you ran into and a few others (line numbering on error messages for example). You can check the luacode package on how to do the catcodes correctly, but if you just wan't to write Lua code, it's not necessary.

Answer 3

By using \beginblock to wrap the catcodes and \endblock to restore them before the actual actually being passed to lua. The allows the argument to \lua to be fixed up by the catcodes, but since \endblock happens before the \directlua call, the catcodes are restored so the output from the lua code is using the original catcodes.

% defines a \lua ... \endlua block for executing luacode
\def\lua{%
  %\directlua{print("\noexpand\n")}% Print out a new line character each time a new lua block is used to force starting on a new line(else it may not)
  \bgroup
  \begingroup
  \catcode`\\=12
  \catcode`\{=12
  \catcode`\}=12
  \catcode`\^^M=12
  \catcode`\#=12
  \catcode`\~=12
  \catcode`\%=12
  \catcode`\_=12
  \catcode`\@=12
  \catcode`\|=0
  \doluacode
}

\bgroup
\catcode`\|=0
\catcode`\^^M=12 %
\catcode`\\=12 %
|long|gdef|doluacode#1^^M#2\endlua{|endgroup|directlua{#2}|egroup}%
|egroup