# Included lilypond files being processed by lualatex can't contain % symbol comments

My MWE below shows two files working together to create a document containing music using the lyluatex package. Unfortunately, I've observed that if I add any comments to the contents anywhere in the file test.ly, the compilation will fail reporting an error (see log slightly cleaned up below) that leads me to believe there is a problem importing a file with comments which rely on % signs in it. I have observed that comments are fine anywhere in the tex file though.

Module lyluatex Warning: Found something incompatible with fragment
(lyluatex)               (or relative). Setting them to false.
(lyluatex)               on input line 34
lyluatex.lua:841: invalid use of '%' in replacement string
stack traceback:
[C]: in function 'gsub' lyluatex.lua:841: in function 'flatten_content'
lyluatex.lua:1082: in function 'output_filename'
lyluatex.lua:531: in function 'calc_properties'
lyluatex.lua:1088: in function 'process'
[\directlua]:1: in main chunk.
\ly@compilescore ...directlua {ly.score:process()}

l.34 \end{tune}

?


Two reproduce: The following MWE is broken into 2 files:

1 - the tex file

2 - the lilypond source file being included

The code as is will compile and produce the expected pdf if compiled with lualatex (i.e. lualatex.exe" -shell-escape "test.tex")

test.tex

\documentclass[letter,fontsize=12pt]{article}

\usepackage{xparse}         % can exclude expl3, xparse loads it
\usepackage{xpatch}         % can exclude etoolbox, xpatch loads it

\usepackage{luatexbase}
\usepackage[program=lilypond]{lyluatex}

\DeclareDocumentCommand{\tune}{}
{%
\ly
}
\def\endtune
{%
\endly
}

\begin{document}

\begin{tune}
%\version "2.18.2"
\include "test.ly"

first = \transpose c d {\tune}
second = \transpose c b {\tune}
\score {
<<
\unfoldRepeats{\first\second}
>>
\layout{}
}
\end{tune}

\end{document}


test.ly

global = {
\key b \minor
\time 9/8
}

tune = \absolute {
\global
\relative c' { c8 d e f g a b c c }
\bar ":|."
}


test.ly (modified to fail)

global = {
\key b \minor
\time 9/8
}

tune = \absolute {
\global % a comment can't go here
% a comment can't seem to go here either
\relative c' { c8 d e f g a b c c }
\bar ":|."
}


I'd like to explore or learn more about why lualatex is choking on my % when it's an expected latex special character.

• Please tell us what's on line 841 in the file lyluatex.lua. (It looks like this file is loaded by the lyluatex.sty LaTeX package file.) – Mico May 13 '18 at 1:53
• @Mico, the lyluatex.lua is a package file, It contains an if statement block: if f then ly_code = ly_code:gsub(iline, self:flatten_content(f:read('*a'))) f:close() end  – EngBIRD May 13 '18 at 1:54
• @Mico, I wonder if this is a regex problem... – EngBIRD May 13 '18 at 2:01
• Lua doesn't actually perform regex. Instead, it does something that's closely related, called pattern matching. (To a first approximation, pattern matching is a subset of regex.) The lua function gsub is one of the functions that performs pattern matching (and substitution). In Lua, the % has a special meaning when pattern matching is performed; e.g., %s- means "zero or more instances of whitespace". To process a % character as itself, it's necessary to input it as %%. Check if replacing single instances of % in test.ly with %% lets the program compile without errors. – Mico May 13 '18 at 2:09
• @Mico, Thanks that worked. Your explanation was really clear, if you want to post it as an answer I will of course upvote and accept. – EngBIRD May 13 '18 at 2:13

The Lua error message

lyluatex.lua:841: invalid use of '%' in replacement string


suggests that something went wrong during a pattern matching operation. (Lua doesn't actually perform regex. Instead, it does something that's closely related, called pattern matching. To a first, crude, approximation, pattern matching is a subset of regex.)

Sure enough, line 841 in the file lyluatex.lua contains the following code:

        ly_code = ly_code:gsub(iline, self:flatten_content(f:read('*a')))


gsub (short for string.gsub) performs pattern matching (and substitution).

In Lua, the % character has a special meaning when it's encountered during pattern matching operations. E.g., %s means "[1 instance of] whitespace character", and %s- means "zero or more instances of whitespace characters". To process a % character as itself, it's necessary to input it as %%.

On to your file test.ly, which is read in by the function that contains line 841 shown above. To make its contents acceptable to Lua's pattern matching operations, it's necessary to double all instances of %:

global = {
\key b \minor
\time 9/8
}

tune = \absolute {
\global %% a comment can now go here
%% a comment can now go here as well
\relative c' { c8 d e f g a b c c }
\bar ":|."
}


Of course, this doesn't guarantee that all will be well; it only guarantees that Lua won't choke on "%". The code in test.ly seems to be some kind of hybrid code; it's probably not a good idea to include comments in it. By the way, to initiate a comment in "pure" Lua code, write --.

This bug has been fixed in the source available on GitHub, but is not available in the TeXLive 2018 distribution. To 'patch' your installation of lyluatex, there are 2 files that need to be copied, as described in the README.

Putting this together (from memory!):

# Get lyluatex from GitHub
$cd ~/Tools$ git clone https://github.com/jperon/lyluatex.git

# Install a fresh version of lyluatex in your usertree
$tlmgr init-usertree$ tlmgr --usermode install lyluatex

# This newly installed lyluatex is here:
$cd ~/texmf/ # Patch it with the 2 files from the GitHub checkout$ cp ~/Tools/lyluatex/lyluatex.lua scripts/lyluatex/lyluatex.lua
\$ cp ~/Tools/lyluatex/lyluatex.sty tex/luatex/lyluatex/lyluatex.sty


I expect this answer will have quite a short shelf-life, and that soon the fixed lyluatex will be available with the main TeXLive distribution.