Escape a string printed from Lua using ConTeXt

Question

This seems to have come up a couple times, but I cannot glean an answer to the generic question of how one might protect an arbitrary string in Lua to ensure its characters are printed.

In other words, given some code in a .lua file:

local i_can_be_anything = '&\\%*_'
context(i_can_be_anything)

This balks for any number of reasons, because the characters have special meaning in TeX when ConTeXt pushes them out to the TeX compiler.

The most apparent option to me would be to convert the string in Lua so that it contains the proper escaping or substitutions that can be passed to TeX to achieve the desired outcome, which would be in this case along the lines of:

\&\textbackslash{}\%\&\_

I believe the full list of replacements one would be looking for would be along the lines of (and this is from some Python I wrote):

tex_replacements = [
    (u'{', ur'\{'),
    (u'}', ur'\}'),
    (u'[', ur'{[}'),
    (u']', ur'{]}'),
    (u'\\', ur'\textbackslash{}'),
    (u'$', ur'\$'),
    (u'%', ur'\%'),
    (u'&', ur'\&'),
    (u'#', ur'\#'),
    (u'_', ur'\_'),
    (u'~', ur'\textasciitilde{}'),
    (u'^', ur'\textasciicircum{}'),
]

tex_mapping =  {ord(char):rep for char, rep in tex_replacements}
def escape(s):
    return unicode(s).translate(tex_mapping)

Is there no function in Lua or LuaTeX to achieve this set of substitutions (or otherwise) so that an arbitrary string could be printed?

A straightforward iteration over the characters with a substitution along the lines of the tex_replacements above could do the trick, but I would expect that someone had already written a solution to this problem.

---

EDIT

It is worth noting that @phg posed a plausible solution of changing the catcodes to vrbcatcode (i.e. \verbatim, or context.sprint(catcodes.numbers.vrbcatcodes, i_can_be_anything)). This does perform the escaping desired, however it seems to have serious issues with line breaking that make it essentially unusable.

Answer 1

This answer is based on an email from Hans Hagen. One way to achieve the desired output is using formatters. For example:

\starttext
\startluacode
  local text='&\%*_'
  context("%!tex!", text)
\stopluacode
\stoptext

The %!tex! is plug to the formatters and tex is one of the pre-defined formatters. Other pre-defined formatters are lua and xml (which escape strings for Lua and XML, respectively).

The above code is equivalent to

context(string.formatters["%!tex!"](text))

which is equivalent to the rather low-level

context(lpeg.match(lpeg.patterns.texescape,text))

The corresponding lpeg patterns for lua and xml formatters are lpeg.patterns.luaescape and lpeg.patterns.xmlescape. These are defined in util-str.lua as:

patterns.xmlescape = Cs((P("<")/"&lt;" + P(">")/"&gt;" + P("&")/"&amp;" + P('"')/"&quot;" + P(1))^0)
patterns.texescape = Cs((C(S("#$%\\{}"))/"\\%1" + P(1))^0)
patterns.luaescape = Cs(((1-S('"\n'))^1 + P('"')/'\\"' + P('\n')/'\\n"')^0) -- maybe also \0
patterns.luaquoted = Cs(Cc('"') * ((1-S('"\n'))^1 + P('"')/'\\"' + P('\n')/'\\n"')^0 * Cc('"'))

The luaquoted pattern is not used in the core.

Answer 2

I am summarizing the comments here, but the gist is:

@phg suggested that one can escape by using catcodes. There is the verbatim catcode, but it has issues with linebreaks.

An alternative, albeit undocumented but seemingly preferable (i.e. escapes without wrapping issues), catcode is \notcatcodes.

One can employ this in Lua to escape any string with the following, for example:

context.sprint(catcodes.numbers.notcatcodes, str)
-- or --
context.pushcatcodetable()
context.setcatcodetable(catcodes.numbers.vrbcatcodes)
context(str)
context.popcatcodetable()