# Problem caching files, aux data not importing

The following code caches include. This negates the need to use \includeonly. The problem is, for references, counters, toc, to be correct, the corresponding aux file needs to be imported. Unfortunately, I can't seem to get it to do anything correct(and I have tried many different ways).

Basically when I manually copy and paste the aux file contents to the tex file then compile, I get the desired results, but when I do the equivalent through lua, I get nothing. It is if the inserted tex is completely ignored.

The project is somewhat complex but I will break it down because some people seem to be easily confused or jump to simple conclusion:

1. This requires windows. Slight modifications should allow it to work on linux or other. The main issue is the hoops required to jump through to do simple file manipulations such as copying due to how poorly TeX is designed for such things.

2. All the following files go in same directly

TeXCache.lua - Main Lua caching code

if __INCLUDES__TeXCache ~= nil then
else
__INCLUDES__TeXCache = true
if __DEBUG == true then
print("Including TeXCache.lua")
end

require "md5"
require "lfs"
require "persist"

------------------------------------------ Utility functions
if not file then file = {} end
function file.Exists(name) local f = io.open(name, "r") if f ~= nil then io.close(f) return true end return false end
function FileExists(name) local f = io.open(name, "r") if f ~= nil then io.close(f) return true end return false end
function FileSize(name) local f = io.open(name, "r") if f == nil then return -1 end local current = f:seek() local size = f:seek("end") f:seek("set", current) io.close(f) return size end

function split(str, pat) local t = {} local fpat = "(.-)" .. pat local last_end = 1 local s, e, cap = str:find(fpat, 1) while s do if s ~= 1 or cap ~= "" then table.insert(t,cap) end last_end = e+1 s, e, cap = str:find(fpat, last_end) end if last_end <= #str then cap = str:sub(last_end) table.insert(t, cap) end return t end
function string.trim(s, c) if not c then c = " \t" end return s:match('^['..c..']*(.-)['..c..']*$') end function string.split(str, sep, n) local sep, fields = sep or ":", {} --local pattern = string.format("([^%s]+)", sep) local pattern = '[^'..sep..']' if n == 0 or n == nil then pattern = '('..pattern..'+)' end if n > 0 then pattern = '^'..(pattern..'+'):rep(n)..'('..pattern..'+)' end str:gsub(pattern, function(c) fields[#fields + 1] = c end) return fields end function DeepCopyTable(orig) local orig_type = type(orig) local copy if orig_type == 'table' then copy = {} for orig_key, orig_value in next, orig, nil do copy[DeepCopyTable(orig_key)] = DeepCopyTable(orig_value) end setmetatable(copy, DeepCopyTable(getmetatable(orig))) else -- number, string, boolean, etc copy = orig end return copy end function CopyFile(source, dest) local infile = io.open(source, "rb") instr = infile:read("*a") infile:close() local outfile = io.open(dest, "wb") outfile:write(instr) outfile:close() end function ReadFile(source, text) if not text then text = "rb" else text = "r" end local infile = io.open(source, text) if not infile then return nil end instr = infile:read("*a") infile:close() return instr end function WriteFile(source, data) local outfile = io.open(source, "wb") instr = infile:write(data) outfile:close() end function DeleteFile(Filename) -- windows only. os.remove has problems!? os.execute('del /Q "'..Filename:gsub("/","\\")..'" > nul') end --------------------------------------------- Setup TeXCache = {} TeXCache.Arguments = ""; TeXCache.ModeAfter = "\\batchmode" TeXCache.Disabled = false TeXCache.FileExt = "_TeXCache.tex" TeXCache.CacheDir = "TeXCache/" TeXCache.HistoryFilename = "TeXCacheHistory.dat"; TeXCache.Header = "" TeXCache.ForceRegenerate = false TeXCache.Count = 0 TeXCache.History = {} TeXCache.Temp = {} --------------------- Preprocessing TeX function StartProcessingTeX() -- Load TeXCache.History and set all values to nil, If they are cached, they will be set to 1 at the end and the nil values keys/files can be removed TeXCache.History = persistence.load(TeXCache.Path..TeXCache.CacheDir..TeXCache.HistoryFilename); if TeXCache.History then if not TeXCache.History.Cached then TeXCache.History.Cached = {} end else TeXCache.History = {} TeXCache.History.Cached = {} end local curPage = -1 local line -- Read the synctex file and attempt to match input file with pages, onces the pages that the input file spans is determined, TeXCache.Temp.ValidCache = false TeXCache.Temp.Synctex = {} TeXCache.Temp.Synctex.Input_ShortFiles = {} TeXCache.Temp.Synctex.Input_Indexes = {} -- Read Synctex If exists, else cannot cache. Stores page relations to input file relations if FileExists(TeXCache.Path..TeXCache.JobName..".synctex") then local f = io.open(TeXCache.Path..TeXCache.JobName..".synctex", "r") if f ~= nil then while true do line = f:read() if line == nil then break end -- Get Input files and store in three way lookup if (string.sub(line,1,6) == "Input:") then local idx = tonumber(split(line, ":")[2]) line = string.sub(line,8 + #tostring(idx),-1) line = string.gsub(string.gsub(line, "\\./", "\\"), "/", "\\") local files = split(line, "\\") local file = files[#files] local file2 = split(file, "%.") -- remove tex extension as it is default(we assume that inputs must have an extension of some type, else collisions may occur) if file2[#file2] == "tex" then file2[#file2] = nil if #file2 > 1 then file2 = table.concat(file2, ".") else file2 = file2[#file2] end file = file2 end TeXCache.Temp.Synctex.Input_ShortFiles[file] = {} TeXCache.Temp.Synctex.Input_ShortFiles[file].idx = idx TeXCache.Temp.Synctex.Input_ShortFiles[file].file = file TeXCache.Temp.Synctex.Input_ShortFiles[file].min = 100000 TeXCache.Temp.Synctex.Input_ShortFiles[file].max = -1 TeXCache.Temp.Synctex.Input_Indexes[idx] = {} TeXCache.Temp.Synctex.Input_Indexes[idx].idx = idx TeXCache.Temp.Synctex.Input_Indexes[idx].file = file TeXCache.Temp.Synctex.Input_Indexes[idx].min = 100000 TeXCache.Temp.Synctex.Input_Indexes[idx].max = -1 elseif (string.sub(line,1,1) == "{") then curPage = string.sub(line,2,-1); elseif #line > 1 then local b = string.sub(line,1,1) if b == "[" or b == "(" or b == "v" or b == "h" or b == "x" or b == "k" or b == "g" or b == "$" then
local idx = tonumber(split(string.sub(line,2,-1), ",")[1])
if not TeXCache.Temp.Synctex.Input_Indexes[idx] then TeXCache.Temp.Synctex.Input_Indexes[idx] = {} end
local lp = math.min(TeXCache.Temp.Synctex.Input_Indexes[idx].min or 100000, curPage)
local hp = math.max(TeXCache.Temp.Synctex.Input_Indexes[idx].max or -1, curPage)
local file = TeXCache.Temp.Synctex.Input_Indexes[idx].file
if file then
TeXCache.Temp.Synctex.Input_Indexes[idx].min = lp
TeXCache.Temp.Synctex.Input_Indexes[idx].max = hp

TeXCache.Temp.Synctex.Input_ShortFiles[file].min = lp
TeXCache.Temp.Synctex.Input_ShortFiles[file].max = hp
end
end
end
end
TeXCache.Temp.Synctex.Exists = true
end
f:close()
end

end

function PreBeginDocument()
end

function TocWriteAux()
-- Import aux file preambles headers
for key, value in pairs(TeXCache.History.Cached) do
if s then
local loc = string.find(s, "\\@setckpt{")
s = s:sub(1,loc-1)
tex.print("\\makeatletter")
tex.print(s)
tex.print("\\makeatother")
print("\n\n\n\r\n\\makeatletter\r\n"..s.."\r\n\\makeatother\r\n\n\n\n")
end
end
end

--------------------- End Of Processing -------------------------
function DoneProcessingTeX()

persistence.store(TeXCache.Path..TeXCache.CacheDir..TeXCache.HistoryFilename, TeXCache.History);
--print("\nTABLE----------------------------"); print(inspect(TeXCache.History)); print("\nTABLE----------------------------------");
print("\n----Total Includes Cached = "..TeXCache.Count.."\n")

if TeXCache.Backups then
for key, value in pairs(TeXCache.Backups) do
-- Create backup of pdf for later use, could optimize duplicate files away
-- We must redirect the copy to a batch file asychronously because TeX people suck ass(too lame)
--CopyFile(TeXCache.Path..TeXCache.JobName..".pdf", TeXCache.Path..TeXCache.CacheDir.."Cached_"..key..".pdf")
if value == 1 then
os.execute('Start "copy" /min '..TeXCache.Path..'CopyFile.bat "'..TeXCache.Path..TeXCache.JobName..".pdf"..'" "'..TeXCache.Path..TeXCache.CacheDir.."Cached_"..key..'.pdf"')
os.execute('Start "copy" /min '..TeXCache.Path..'CopyFile.bat "'..TeXCache.Path..key..".aux"..'" "'..TeXCache.Path..TeXCache.CacheDir.."Cached_"..key..'.aux"')
end
end
end

-- Another hoop to jump through because of the ignorant TeX people. syntex(busy) won't be renamed when using os.execute above for some ignorant reason
os.execute('Start "copy" /min '..string.gsub('CopyFile "'..TeXCache.Path..TeXCache.JobName..'.synctex(busy)" "'..TeXCache.Path..TeXCache.JobName..'.synctex"', "/", "\\"))
end

-- Function callbacks must be above

-------------------------------------- Caching code

function TeXCache.Initialize(JobName, path)
-- To write aux files without getting strange write error one must set MIKTEX_ALLOWUNSAFEOUTPUTFILES=1 in environment for miktex.
-- Get base_dir using dos if necessary
if not path then local p = io.popen("echo %cd%") if p then path = p:read("*l"):trim("\\").."\\" p:close() end end
if path == nil or path == "" then path = lfs.currentdir() end
path = (path or ""):trim('"\' \\/').."/"
JobName = JobName:trim('"\' /') or "temp"
TeXCache.JobName = JobName
TeXCache.Path = path
print('Path = '..TeXCache.Path)

StartProcessingTeX()
end tex.print("\\directlua{TeXCache.Initialize('\\jobname', '\\BasePath')}")

function TeXCache.Include(code)

-- compare hash of included file with previously stored hash, if the same, then use cache
-- could use filesize/date comparision for quicker tests
local codefilename = TeXCache.Path..code..".tex"
local fs = -1
local hash = -1
local curIdx = -1
local lp = -1
local hp = -1

if FileExists(codefilename) then
fs = FileSize(codefilename)
if  TeXCache.History.Cached[code] and TeXCache.History.Cached[code].FileSize == fs then
if TeXCache.History.Cached[code].Hash == hash then
local idx = -1
if TeXCache.Temp.Synctex.Input_ShortFiles[code] then
curIdx = TeXCache.Temp.Synctex.Input_ShortFiles[code].idx
lp = math.max(TeXCache.Temp.Synctex.Input_ShortFiles[code].min, TeXCache.History.Cached[code].LowPage)
hp = math.max(TeXCache.Temp.Synctex.Input_ShortFiles[code].max, TeXCache.History.Cached[code].HighPage)
else
lp = TeXCache.History.Cached[code].LowPage
hp = TeXCache.History.Cached[code].HighPage
end
if lp < 0 or hp < 0 or lp >= 99999 then
TeXCache.Temp.ValidCache = false
else
TeXCache.Temp.ValidCache = true
end
else
TeXCache.Temp.ValidCache = false
end
else
TeXCache.Temp.ValidCache = false
end
else
TeXCache.Temp.ValidCache = false
end

if TeXCache.Temp.ValidCache == false then
tex.print("\\include{"..code.."}")
if TeXCache.Backups == nil then
TeXCache.Backups = {}
end
TeXCache.Backups[code] = 1
-- Store Cached information
if FileExists(codefilename) then
fs = FileSize(codefilename)
end
if TeXCache.History.Cached[code] then
fs = fs
hash = hash
curIdx = curIdx
lp = math.max(lp, TeXCache.History.Cached[code].LowPage or -1);
hp = math.max(hp, TeXCache.History.Cached[code].HighPage or -1);
end
TeXCache.History.Cached[code] = {}
TeXCache.History.Cached[code].FileSize = fs
TeXCache.History.Cached[code].Hash = hash
TeXCache.History.Cached[code].LowPage = lp
TeXCache.History.Cached[code].HighPage = hp
return
end

for key, value in pairs(TeXCache.History.Cached) do
if code == key then
print("\nUsing cached "..code..".tex!\n")
local filename = TeXCache.Path..TeXCache.CacheDir.."Cached_"..code
tex.print("\\includepdf[pages={"..lp.."-"..hp.."}]{"..filename.."}")

-- Import aux file counters
if s then
local _, loc = string.find(s, "\\@setckpt{"..code.."}{")
s = s:sub(loc+1, -1)
for i=#s, 1, -1 do
if s:sub(i,i) == "}" then
s = string.sub(s,1,i-1)
break;
end
end

s = "\r\n\\makeatletter\r\n"..s.."\r\n\\makeatother\r\n"
tex.print(s)
print("\r\n\r\n\r\n"..s.."\r\n\r\n\r\n")
end

-- Store cached info for use on next run
TeXCache.History.Cached[code].FileSize = fs
TeXCache.History.Cached[code].Hash = hash
TeXCache.History.Cached[code].LowPage = lp
TeXCache.History.Cached[code].HighPage = hp
TeXCache.Count = TeXCache.Count + 1
break
end
end

end

end -- __INCLUDE__


CopyFile.bat - Used to copy files asynchronously

@echo off
@REM use asynchronously to copy a file after a process as exited(set timeout to be long enough for process to end).
@REM Call with Start "copy file" <path to this batch file> <source> <dest>

TIMEOUT 3 /NOBREAK
copy /B /Y %1 %2
exit


Clear.bat - Clears junk files to remove the caching(to start reset everything)

@echo off
del test.synctex(busy)
del test.synctex
del test.pdf
del test.toc
del test.log
del test.aux
del TeXCache\TeXCacheHistory.dat
del TeXCache\Cached_testinput.pdf
del TeXCache\Cached_testinput.aux


persist.lua - Used to store and retrive lua tables(can get from git hub)

--t_original = {1, 2, ["a"] = "string", b = "test", {"subtable", [4] = 2}};
--persistence.store("storage.lua", t_original);

local write, writeIndent, writers, refCount;

persistence =
{
store = function (path, ...)
local file, e = io.open(path, "w");
if not file then
return error(e);
end
local n = select("#", ...);
-- Count references
local objRefCount = {}; -- Stores reference that will be exported
for i = 1, n do
refCount(objRefCount, (select(i,...)));
end;
-- Export Objects with more than one ref and assign name
-- First, create empty tables for each
local objRefNames = {};
local objRefIdx = 0;
file:write("-- Persistent Data\n");
file:write("local multiRefObjects = {\n");
for obj, count in pairs(objRefCount) do
if count > 1 then
objRefIdx = objRefIdx + 1;
objRefNames[obj] = objRefIdx;
file:write("{};"); -- table objRefIdx
end;
end;
file:write("\n} -- multiRefObjects\n");
-- Then fill them (this requires all empty multiRefObjects to exist)
for obj, idx in pairs(objRefNames) do
for k, v in pairs(obj) do
file:write("multiRefObjects["..idx.."][");
write(file, k, 0, objRefNames);
file:write("] = ");
write(file, v, 0, objRefNames);
file:write(";\n");
end;
end;
-- Create the remaining objects
for i = 1, n do
file:write("local ".."obj"..i.." = ");
write(file, (select(i,...)), 0, objRefNames);
file:write("\n");
end
-- Return them
if n > 0 then
file:write("return obj1");
for i = 2, n do
file:write(" ,obj"..i);
end;
file:write("\n");
else
file:write("return\n");
end;
if type(path) == "string" then
file:close();
end;
end;

local f, e;
if type(path) == "string" then
else
end
if f then
return f();
else
return nil, e;
end;
end;
}

-- Private methods

-- write thing (dispatcher)
write = function (file, item, level, objRefNames)
writers[type(item)](file, item, level, objRefNames);
end;

-- write indent
writeIndent = function (file, level)
for i = 1, level do
file:write("\t");
end;
end;

-- recursively count references
refCount = function (objRefCount, item)
-- only count reference types (tables)
if type(item) == "table" then
-- Increase ref count
if objRefCount[item] then
objRefCount[item] = objRefCount[item] + 1;
else
objRefCount[item] = 1;
-- If first encounter, traverse
for k, v in pairs(item) do
refCount(objRefCount, k);
refCount(objRefCount, v);
end;
end;
end;
end;

-- Format items for the purpose of restoring
writers = {
["nil"] = function (file, item)
file:write("nil");
end;
["number"] = function (file, item)
file:write(tostring(item));
end;
["string"] = function (file, item)
file:write(string.format("%q", item));
end;
["boolean"] = function (file, item)
if item then
file:write("true");
else
file:write("false");
end
end;
["table"] = function (file, item, level, objRefNames)
local refIdx = objRefNames[item];
if refIdx then
-- Table with multiple references
file:write("multiRefObjects["..refIdx.."]");
else
-- Single use table
file:write("{\n");
for k, v in pairs(item) do
writeIndent(file, level+1);
file:write("[");
write(file, k, level+1, objRefNames);
file:write("] = ");
write(file, v, level+1, objRefNames);
file:write(";\n");
end
writeIndent(file, level);
file:write("}");
end;
end;
["function"] = function (file, item)
-- Does only work for "normal" functions, not those
-- with upvalues or c functions
local dInfo = debug.getinfo(item, "uS");
if dInfo.nups > 0 then
file:write("nil --[[functions with upvalue not supported]]");
elseif dInfo.what ~= "Lua" then
file:write("nil --[[non-lua function not supported]]");
else
local r, s = pcall(string.dump,item);
if r then
else
file:write("nil --[[function could not be dumped]]");
end
end
end;
end;
["userdata"] = function (file, item)
file:write("nil --[[userdata]]\n");
end;
}


test.tex - The demo tex file to test

\documentclass[12pt,oneside]{book}
%\batchmode

\directlua{
require("debugger")()
}

\directlua{
% Must be ran first, sets up paths
__DEBUG = true
local path = BasePath
local p = io.popen('echo '..string.char(37)..'cd'..string.char(37)) if p and (path == nil or path == "") then path = p:read("*l") p:close() end
if path == nil or path == "" then path = lfs.currentdir() end
if path == nil or path == "" then else BasePath = path end
path = string.gsub(path, string.char(92), "/");
BasePath = string.gsub(BasePath, string.char(92), "/")
tex.print(-1, string.char(92).."def"..string.char(92).."BasePath"..string.char(123)..BasePath..string.char(125))
}

\directlua{
dofile('TeXCache.lua')
}

\usepackage{luatex}
\usepackage{xcolor}
\usepackage{everypage}
\usepackage{afterpage}
\usepackage{pdfpages}
\usepackage{everyhook}
\usepackage{ifthen}
\usepackage{letltxmacro}
\usepackage{atbegshi}
\usepackage{lipsum}

\directlua{tex.enableprimitives('',tex.extraprimitives())}

\newcommand{\tableofcontentsEx}{\tableofcontents\directlua{TocWriteAux()}}
\newcommand{\cachedInput}[1]{\clearpage\directlua{TeXCache.Include("\luaescapestring{#1}"); }\clearpage}

\title{Sections and Chapters}
\author{Test Author}
\date{ }

\directlua{if PreBeginDocument then PreBeginDocument() end}
\begin{document}

\maketitle
\tableofcontentsEx
\clearpage

\lipsum

%This either includes or includespdf when cached.
\cachedInput{testinput}

slfkljasfasfjllasfjas;df

234214214892784
\chapter{HELP!!!}
asdfjhajsdfkljasjdf
a
asdffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff

\end{document}


testinput.tex - the demo input text file - can be any normal tex file

\chapter{AMAZING!}

{{{}}}{}{}{}{}{}{}
$$x = y + z + z^2 + z^{311}$$

\begin{tabular}{|c|c|}
\hline
% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
1 & 1 \\
2 & 2 \\
\hline

\end{tabular}

$$x = y$$

$z = x$

A\lipsum

\{

\textcolor{red}{
This is a test cache1
B\lipsum
\textcolor{green}{C\lipsum}
}

$$x = y + z + z^2 + z^43434$$
\lipsum
\lipsum
\lipsum

$z = x$

D\lipsum
test
xxxxxxxxxxxxxxxx


These are all the files one needs. One can find the full project at http://www.filedropper.com/test_192

It would be nice if the code specified line numbers so I could reference the blocks that deal with the aux file. In any case, There are only really two parts in the TeXCache.lua that deal with the aux.

One is to try to load it in the preamble using TocWriteAux() and the other is in the TeXCache.Include function. This occurs where it says "-- Import aux file counters".

What I tried to do was split the aux file up in to what occurs before setckpt and what occurs after because simply using '\input' did not work.

One can remove the aux code and empty TocWriteAux() to not have any aux behavior from the lua code. I only included it so one could potentially modify it if there is an issue.

Again, the main issue is that when I "tex.print" the aux code in lua it doesn't work yet it works when I copy and paste. Why? I have no idea! That is why I am asking. Please don't "help" with generic answers but by providing a solution that makes the test project work as expected(any answers that do not solve this problem are not answers. It should also work in general(at least 95% of test cases.)). I've went out of my way to provide a through "MWE" and I expect anyone trying to help to do the same rather than trying to get a "gold star" or whatever.

So, what is the expected behavior?

1. The toc is always correct. I realize that the way the toc is that it might require an extra compilation for it to update properly but that is a general issue with the toc design and not the caching mechanism.

2. The chapter counters are always right. They are right after the first run but when the caching mechanism kicks in, they are wrong because the aux file is not being used. Other counters and such should also be properly imported. The whole point of the aux file is to keep such things in sync, and so it should behave as it should regardless of the caching design.

e.g., after compiling test a few times, The "Help" chapter is 1 when it should be 2. One can insert the aux file directly(copy and paste_ and things work. So why the lua code, which essentially does the same thing(at least what I tried first before I tried to split), does not work is beyond me.

1. The lua code should not be necessary to import the aux info? I have been told this but the methods I tried did not work. e.g., after the line

tex.print("\includepdf[pages={"..lp.."-"..hp.."}]{"..filename.."}")

tex.print("\\input{"..filename..".aux}")
`

and it should import the aux file, but it does nothing. Just as if one added it directly to the tex.

Anyways, hopefully someone will put in the time to get this to work. Being able to cache includes is a nice feature and quite a speed saver for slow projects. While it is a bit too granular compared to per page caching, it is better than nothing and avoids the "includeonly" bs that we currently use.

I'd expect the answer to be simple as it is just doing something that tex already does... e.g., include the aux file. But either I am screwing that up or it simply is not working as it should(a bug or due to some weird issue with the TeXCache code).

Note, another bug is that the synctex file is not closed properly. Why that occurs seems to be another bug. It occurs only because I delegate the copying of the pdf file asynchronously to the os. This is because there is no post pdf callback. If one tries to copy the pdf at the "end" through lua(either the stop_run callback or whatever) the pdf is not completely finished and is invalid. These issues and workarounds are due to the cobbled nature of TeX, LuaLatex, and syntex. If a proper callback hierarchy was added along with a few other things some very good features could be created. (e.g., nearly a transparent full page by page caching mechanisms, etc)

Anyways, thanks for any help...

• And this solution should work in general, not just for the demo, obviously. – AbstractDissonance Mar 2 '17 at 1:14

## protected by Joseph Wright♦Mar 7 '17 at 13:49

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).