Included lilypond files being processed by lualatex can't contain % symbol comments

Question

My MWE below shows two files working together to create a document containing music using the lyluatex package. Unfortunately, I've observed that if I add any comments to the contents anywhere in the file test.ly, the compilation will fail reporting an error (see log slightly cleaned up below) that leads me to believe there is a problem importing a file with comments which rely on % signs in it. I have observed that comments are fine anywhere in the tex file though.

Module lyluatex Warning: Found something incompatible with `fragment`
(lyluatex)               (or `relative`). Setting them to false.
(lyluatex)               on input line 34
lyluatex.lua:841: invalid use of '%' in replacement string
stack traceback: 
    [C]: in function 'gsub' lyluatex.lua:841: in function 'flatten_content'
    lyluatex.lua:1082: in function 'output_filename' 
    lyluatex.lua:531: in function 'calc_properties'
    lyluatex.lua:1088: in function 'process'
    [\directlua]:1: in main chunk.
\ly@compilescore ...directlua {ly.score:process()}

l.34 \end{tune}

? 

Two reproduce: The following MWE is broken into 2 files:

1 - the tex file

2 - the lilypond source file being included

The code as is will compile and produce the expected pdf if compiled with lualatex (i.e. lualatex.exe" -shell-escape "test.tex")

test.tex

\documentclass[letter,fontsize=12pt]{article}

\usepackage{xparse}         % can exclude expl3, xparse loads it
\usepackage{xpatch}         % can exclude etoolbox, xpatch loads it

\usepackage{luatexbase}
\usepackage[program=lilypond]{lyluatex}

\DeclareDocumentCommand{\tune}{}
{%
    \ly 
}
\def\endtune
{%
    \endly
}

\begin{document}

\begin{tune}
    %\version "2.18.2"
    \include "test.ly"

    first = \transpose c d {\tune}
    second = \transpose c b {\tune}
    \score {
        <<
        \unfoldRepeats{\first\second}       
        >>
        \layout{}
    }
\end{tune}

\end{document}

test.ly

global = {
\key b \minor
\time 9/8
}

tune = \absolute {
\global
\relative c' { c8 d e f g a b c c }
\bar ":|."
}

test.ly (modified to fail)

global = {
\key b \minor
\time 9/8
}

tune = \absolute {
\global % a comment can't go here
% a comment can't seem to go here either
\relative c' { c8 d e f g a b c c }
\bar ":|."
}

I'd like to explore or learn more about why lualatex is choking on my % when it's an expected latex special character.

Answer 1

The Lua error message

lyluatex.lua:841: invalid use of '%' in replacement string

suggests that something went wrong during a pattern matching operation. (Lua doesn't actually perform regex. Instead, it does something that's closely related, called pattern matching. To a first, crude, approximation, pattern matching is a subset of regex.)

Sure enough, line 841 in the file lyluatex.lua contains the following code:

        ly_code = ly_code:gsub(iline, self:flatten_content(f:read('*a')))

gsub (short for string.gsub) performs pattern matching (and substitution).

In Lua, the % character has a special meaning when it's encountered during pattern matching operations. E.g., %s means "[1 instance of] whitespace character", and %s- means "zero or more instances of whitespace characters". To process a % character as itself, it's necessary to input it as %%.

On to your file test.ly, which is read in by the function that contains line 841 shown above. To make its contents acceptable to Lua's pattern matching operations, it's necessary to double all instances of %:

global = {
\key b \minor
\time 9/8
}

tune = \absolute {
\global %% a comment can now go here
%% a comment can now go here as well
\relative c' { c8 d e f g a b c c }
\bar ":|."
}

Of course, this doesn't guarantee that all will be well; it only guarantees that Lua won't choke on "%". The code in test.ly seems to be some kind of hybrid code; it's probably not a good idea to include comments in it. By the way, to initiate a comment in "pure" Lua code, write --.

Answer 2

This bug has been fixed in the source available on GitHub, but is not available in the TeXLive 2018 distribution. To 'patch' your installation of lyluatex, there are 2 files that need to be copied, as described in the README.

Putting this together (from memory!):

# Get lyluatex from GitHub
$ cd ~/Tools
$ git clone https://github.com/jperon/lyluatex.git

# Install a fresh version of lyluatex in your usertree
$ tlmgr init-usertree
$ tlmgr --usermode install lyluatex

# This newly installed lyluatex is here:
$ cd ~/texmf/

# Patch it with the 2 files from the GitHub checkout
$ cp ~/Tools/lyluatex/lyluatex.lua scripts/lyluatex/lyluatex.lua 
$ cp ~/Tools/lyluatex/lyluatex.sty tex/luatex/lyluatex/lyluatex.sty 

I expect this answer will have quite a short shelf-life, and that soon the fixed lyluatex will be available with the main TeXLive distribution.