# Weird double processing issue with process_input_buffer

I am having problems with using process_input_buffer to count lines. It seems that \includes are expanded and processed multiple times. I believe it is specific to \include

1. I store and print out the source code using process_input_buffer.

2. I have a macro that writes its argument to a file then uses \include to include that data(or a pdf, which I haven't added).

The problem: When I use this macro I end I seeing the output of the macro twice. One from it just being part of the input file and the other from being included by the macro. I do not ever actually see the \include line, just the results.

It seems that process_input_buffer expands the macro after it is called while I need it before so I don't double process any lines in the source code(since I only care about the source code and not the "input buffer" per se). Now, it is effectively doing what it wants but I cannot write a source to pdf mapper the way it is.

Without such macros I can map any source line in the document environment to its pdf page. I imagine this is not robust though as other weird processing events will cause similar problems.

What I would like is a process_source_buffer(the source code read before any modifications, and it is never different than what is) but I guess I'm not that lucky.

Alternatively, I am thinking that I might have to read the file manually and try to stitch it up with process_input buffer but this doesn't seem very effective as some cases where duplicate lines are used might create a matching problem. i.e., every line the the source file appears at least once in the input buffer but may appear more than once and some macro modifications might trigger invalid matching throwing the line numbering off.

e.g., if a macro happens to include the same lines of course code that occurs in the future in the actual source code, the matching will think it is a source line and label its line value wrong. This is the only solution I can come up with and should work for many cases but is not robust and no desirable. I know LuaLatex is quite limited in its capabilities but surely there is a way to process the pre-expanded tex?

e.g.,

Alpha
\includeMe{Beta}
Beta


and \includeMe just writes Beta to a file then includes it, there is no way to know that the 2nd beta is the real beta. Because what process_input_buffer sees is

Alpha
\includeMe{Beta}
Beta
Beta


and the 2nd beta is actually the one included from the include file and will trigger a source code matching(since everything looks identical up to that point with the original) and the line numbering I'm trying to create will be off.

I don't ever see that really happening but since it is possible my method may fail.

Any ideas?

(would be tremendously easier if LuaLatex just had these features built in)

• @DavidCarlisle No, it is not, I have proof. I wouldn't have posted this if that was the case. You fail to realize that it only does the current line, but if you "modify"(from a macro) anything after that line, the modifications will be processed. (it has to, which I'm sure you get, but that makes it impossible to know what is actually part of the source file and what is "injected" code) If it is processing line X and that line has a macro on it, then line X + n will end up being the output of that macro. So, it is pre-processing per line, but not per file or project. – AbstractDissonance Feb 10 '17 at 0:05
• Again, It may be pre-expansion of the current processed line, but the next line, or some line after that will potentially contain text that is not in the actual file. This is fine, although unclear as no distinction is made by the documentation and there is no alternative for what I want. So we are talking about two different things here as usual. (one of the great things about TeX is how confusing and ambiguous it's terminology and style is) – AbstractDissonance Feb 10 '17 at 0:08
• sigh, that callback is hooked in to the file reader and gets the line of raw characters as a string before it is even tokenized so clearly before tex processing. it gets each line read from the file exactly once unless you input the file twice. – David Carlisle Feb 10 '17 at 0:09
• better would be to fix the question to have a usable example! – David Carlisle Feb 10 '17 at 0:10

As in the earlier question, you are not printing any numbers that tie text in the output to page numbers. here you are simply showing the value of the page counter at the point the line is scanned, which is 1 in all cases here.

\documentclass[12pt,oneside]{book}

\directlua{require('TeXCache.lua')}

\begin{document}
%__TeXCache_FILESTART__
\setcounter{page}{12}

One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.
One two three four five six seven eight nine ten.
Red yellow blue green purple black white brown.
Apple orange banana grape pear lemon.

%__TeXCache_FILEEND__
\end{document}


and a log of

Line = 1, Page = 1, Source = \setcounter{page}{12}
Line = 2, Page = 1, Source =
Line = 3, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 4, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 5, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 6, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 7, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 8, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 9, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 10, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 11, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 12, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 13, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 14, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 15, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 16, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 17, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 18, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 19, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 20, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 21, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 22, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 23, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 24, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 25, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 26, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 27, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 28, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 29, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 30, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 31, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 32, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 33, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 34, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 35, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 36, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 37, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 38, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 39, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 40, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 41, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 42, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 43, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 44, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 45, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 46, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 47, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 48, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 49, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 50, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 51, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 52, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 53, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 54, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 55, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 56, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 57, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 58, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 59, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 60, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 61, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 62, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 63, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 64, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 65, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 66, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 67, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 68, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 69, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 70, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 71, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 72, Page = 1, Source = One two three four five six seven eight nine ten.
Line = 73, Page = 1, Source = Red yellow blue green purple black white brown.
Line = 74, Page = 1, Source = Apple orange banana grape pear lemon.
Line = 75, Page = 1, Source =


This just gives the same information that you got from your original tex hook, that no pages are output until tex reads the blank line on what is now line 75.

That is correct information, but it is apparently not the information that you want. It isn't clear what information you _do want.

• I thought we solve the paged counting problem? The whole point of the hook and what you told me was correct, was it not? Are you saying that pages are not generated in any way that correlates to how they are processed? My thinking is that as TeX processes the input buffer and forms the output, when the output is enough to write a page, it does so. This is why I get the page output's intermixed with code. In your example, the pages are not generated until the end. I see that now, but why? Why is your example generating pages only at the end of the document while in my example it is not? – AbstractDissonance Feb 10 '17 at 21:57
• I am not sure how TeX forms a page and generates it. Please tell me when "start_page_number" is called? I thought it was called after enough tokens was collected to generate a pdf page and that pdf page was generated. \newpage seems to trigger it just fine. But your code doesn't have any natural page divisions that I would think it would, e.g., around line 71(in full listing). – AbstractDissonance Feb 10 '17 at 22:02
• @AbstractDissonance I'll post to the existing chat chat.stackexchange.com/rooms/53266/… – David Carlisle Feb 10 '17 at 22:20

If you want to know correlation between the lines in the source and the typeset pages you need \labels. See the example below. But before you now think that this is the solution to your problem: There are various cases where you would get wrong or at least unclear output e.g.

  \label{line:X}\newpage abc

\label{line:X}\section{abc} % when it starts a new page

\label{line:X}\begin{figure} blblb\end{figure} % the float can be everywhere


There are also various cases where you can get horrible errors if you inject a label at the start of line, e.g.

  \label{line:X}\multicolumn{2}{l}{abc}


So I maintain my claim that it will not work as you want it to work.

\documentclass[12pt,oneside]{book}

\usepackage{graphicx}
\usepackage[paperheight=6cm]{geometry}

\usepackage{luacode}
\usepackage{listings}
\begin{document}

\label{line:01}One two three four five six seven eight nine ten.
\label{line:02}Red yellow blue green purple black white brown.
\label{line:03}Apple orange banana grape pear lemon.
\label{line:04}One two three four five six seven eight nine ten.
\label{line:05}Red yellow blue green purple black white brown.
\label{line:06}Apple orange banana grape pear lemon.
\label{line:07}One two three four five six seven eight nine ten.
\label{line:08}Red yellow blue green purple black white brown.
\label{line:09}Apple orange banana grape pear lemon.
\label{line:10}One two three four five six seven eight nine ten.
\label{line:11}Red yellow blue green purple black white brown.
\label{line:12}Apple orange banana grape pear lemon.
\label{line:13}One two three four five six seven eight nine ten.
\label{line:14}Red yellow blue green purple black white brown.
\label{line:15}Apple orange banana grape pear lemon.
\label{line:16}One two three four five six seven eight nine ten.
\label{line:17}Red yellow blue green purple black white brown.
\label{line:18}Apple orange banana grape pear lemon.
\label{line:19}One two three four five six seven eight nine ten.
\label{line:20}Red yellow blue green purple black white brown.
\label{line:21}Apple orange banana grape pear lemon.
\label{line:22}One two three four five six seven eight nine ten.
\label{line:23}Red yellow blue green purple black white brown.
\label{line:24}Apple orange banana grape pear lemon.
\label{line:25}One two three four five six seven eight nine ten.
\label{line:26}Red yellow blue green purple black white brown.
\label{line:27}Apple orange banana grape pear lemon.
\label{line:28}One two three four five six seven eight nine ten.
\label{line:29}Red yellow blue green purple black white brown.
\label{line:30}Apple orange banana grape pear lemon.
\label{line:31}One two three four five six seven eight nine ten.
\label{line:32}Red yellow blue green purple black white brown.
\label{line:33}Apple orange banana grape pear lemon.
\label{line:34}One two three four five six seven eight nine ten.
\label{line:35}Red yellow blue green purple black white brown.
\label{line:36}Apple orange banana grape pear lemon.
\label{line:37}One two three four five six seven eight nine ten.
\label{line:38}Red yellow blue green purple black white brown.
\label{line:39}Apple orange banana grape pear lemon.
\label{line:40}One two three four five six seven eight nine ten.
\label{line:41}Red yellow blue green purple black white brown.
\label{line:42}Apple orange banana grape pear lemon.
\label{line:43}One two three four five six seven eight nine ten.
\label{line:44}Red yellow blue green purple black white brown.
\label{line:45}Apple orange banana grape pear lemon.
\label{line:46}One two three four five six seven eight nine ten.
\label{line:47}Red yellow blue green purple black white brown.
\label{line:48}Apple orange banana grape pear lemon.
\label{line:49}One two three four five six seven eight nine ten.
\label{line:50}Red yellow blue green purple black white brown.
\label{line:51}Apple orange banana grape pear lemon.
\label{line:52}One two three four five six seven eight nine ten.
\label{line:53}Red yellow blue green purple black white brown.
\label{line:54}Apple orange banana grape pear lemon.
\label{line:55}One two three four five six seven eight nine ten.
\label{line:56}Red yellow blue green purple black white brown.
\label{line:57}Apple orange banana grape pear lemon.
\label{line:58}One two three four five six seven eight nine ten.
\label{line:59}Red yellow blue green purple black white brown.
\label{line:60}Apple orange banana grape pear lemon.
\label{line:61}One two three four five six seven eight nine ten.
\label{line:62}Red yellow blue green purple black white brown.
\label{line:63}Apple orange banana grape pear lemon.
\label{line:64}One two three four five six seven eight nine ten.
\label{line:65}Red yellow blue green purple black white brown.
\label{line:66}Apple orange banana grape pear lemon.
\label{line:67}One two three four five six seven eight nine ten.
\label{line:68}Red yellow blue green purple black white brown.
\label{line:69}Apple orange banana grape pear lemon.
\label{line:70}One two three four five six seven eight nine ten.
\label{line:71}Red yellow blue green purple black white brown.
\label{line:72}Apple orange banana grape pear lemon.

\begin{luacode*}

for counter = 1, 72 do
tex.print("line "..counter.." is on page \\pageref{line:"..string.format("%02d",counter).."}\\par")
end
\end{luacode*}
\end{document}


• I am not trying to inject any text. I simply want to know, in lua, the source text line to page mapping. Let me explain VERY CLEARLY and this is fact: For any source code in the text file, it maps to a some place in the pdf or simply doesn't map at all. Agreed? Yes or no. simple procedure to verify: 1. Generate the pdf document from ANY tex file. Open the Pdf and then procedure to find which source lines are on which pages. Some may map to multiple pages, that is ok. Just write them down. I can do this process manually but I need to be able to do this automatically and robustly. – AbstractDissonance Feb 10 '17 at 22:09
• if you don't agree with my confusion that it can be done, demonstrate any .tex file that is a counter example. Of course, not using complexity simply to make it take a long time or just be confusing to me, but something that inherently cannot do what I claim... I will, then attempt to match the source text line withe pdf pages. If I can succeed then your counter example fails. if I cannot produce a listing then I fail and I will shut up about all this. – AbstractDissonance Feb 10 '17 at 22:11
• @AbstractDissonance I think the question you pose in the comment is just not well formed and in general has no answer. a tex source file is input to a program the pdf is its output. In simple cases they both happen to look a bit like text and you may see the same bits of text in each, but in general the output of a program depends on the entire program, you can not say one part of the output depends on one part of the input. – David Carlisle Feb 10 '17 at 23:22
• @DavidCarlisle It might be true theoretically but, at least in all the work I do there is a direct mapping between what I see in the tex and what exits on the pdf. As you mentioned before, syntex solves this problem. If you click somewhere on the pdf it takes you to a line. Even images and things like tables are found. And that is mapping a mouse click coordinate(x,y) to a line and column location which is far more than I need. In fact, I just need a rough estimate of the page breaks, but I do need them well formed. I realize not everything can be cached, but most things can. – AbstractDissonance Feb 10 '17 at 23:29
• The problem is the difficulty in doing this. I could absolute get what I want if I wanted to rewrite a typographical system(re implement the wheel better) but obviously I'm not going to do that and it would not be practical. What it seems like I'm going to have to resort to use littering my sources with the caching macro. I'll have to manually make sure they are well formed and in reasonable chunk sizes to get good and valid caching. Not difficult but I like automated ways better. – AbstractDissonance Feb 10 '17 at 23:32