# Modifying TeX/Synctex

There is a glaring design flaw in TeX or Synctex for location of words.

Given a text in a .tex files:

Blah blah blah. This is a sentence blah! Blah Blah!


The whole paragraph is treated as one unit rather than the individual words.

It turns out that if one adds new lines to each word or line then the syncing works great:

Blah
blah
blah.
This
is
a
sentence
blah!
Blah
Blah!


And we get single word accuracy.

The problem is that it makes reading the tex file pointless and hence it is not a proper solution.

A real solution would be to have the compiler to automatically translate words(or lines) in to separate lines automatically.

This would provide the proper syntex for word or line resolution but the back propagation would not match the original unaltered .tex file.

The line info would need to be kept track of and then inversely mapped.

Is there any way to do this easily?

e.g.,

Line1:
Blah blah blah. This is a sentence blah! Blah Blah!
Line2:


becomes

Line1:
Blah
Line 2:
blah
Line 3:
blah.
Line 4:
This
Line 5:
is
Line 6:
a
Line 7:
sentence
Line 8:
blah!
Line 9:
Blah
Line 10:
Blah!


And these inserted new lines positions are then kept track of to do the inverse map back in to the column that they were inserted from.

This is all basic mapping and pretty easy do to with a few lines of code except for a few problems.

1: One must not insert new lines in invalid place that will break the .tex file or alter it's output. 2: The editor that responds to the synctex must be able to do the inverse mapping from lines to line/col.

Essentially one could insert a new line in every space as long as it won't break the .tex code and would get word level accuracy. This could potentially be done pre-compilation if proper word boundaries could be determined. I think though it would be better if it could be done in an environment such as

\begin{synctexing}
…
\end{synctexing}


which would essentially add the new lines to all the text in between and ignore any environments inside so things like tikz pictures could be inserted without having to create a bunch of environments.

It would simply output the mapping file(line/col to line) and the editor would have to be able to reverse the map(easy but it would require supporting it unless they already support columns(but that synctex just always gives them 0).

The problem is not that this is difficult but about the specifics which I know nothing about.

In fact, even if we had to do something like \snl which adds a new line to the synctex, it would be better than nothing.

• in very rough terms synctex can map word for word using "hinting" and some editors can jump character to character --- However to be universally functional in the majority of cases it tries to show unknown editor line for unknown renderer line so at its simplest its a basic cross index of tex line17 = pdf line4 with all the issues that entails – user170109 Nov 29 '18 at 23:54
• What is “Syntex”? Did you mean SyncTeX? – Henri Menke Nov 29 '18 at 23:55
• SyncTeX does not record glyph nodes but kerns and glues. Since interword spaces are glue nodes, SyncTeX should in principle have single-word granularity. – Henri Menke Nov 30 '18 at 0:06
• Since you have some responses below that seem to answer your question, please consider marking one of them as ‘Accepted’ by clicking on the tickmark below their vote count (see How do you accept an answer?). This shows which answer helped you most, and it assigns reputation points to the author of the answer (and to you!). It's part of this site's idea to identify good questions and answers through upvotes and acceptance of answers. – samcarter_is_at_topanswers.xyz Jan 10 '19 at 19:50

The SyncTeX architecture currently maps the position on the PDF page to the line and a “tag” in the input file (I don't really know what “tag“ is).

This is the SyncTeX data structure from syntex.c:

/*  Here are all the local variables gathered in one "synchronization context"  */
static struct {
void *file;                 /*  the foo.synctex or foo.synctex.gz I/O identifier  */
synctex_fprintf_t fprintf;  /*  either fprintf or gzprintf */
char *busy_name;            /*  the real "foo.synctex(busy)" or "foo.synctex.gz(busy)" name, with output_directory  */
char *root_name;            /*  in general jobname.tex  */
integer count;              /*  The number of interesting records in "foo.synctex"  */
/*  next concern the last sync record encountered  */
halfword node;              /*  the last synchronized node, must be set
*  before the recorder */
synctex_recorder_t recorder;/*  the recorder of the node above, the
*  routine that knows how to record the
*  node to the .synctex file */
integer tag, line;          /*  current tag and line  */
integer curh, curv;         /*  current point  */
integer magnification;      /*  The magnification as given by \mag */
integer unit;               /*  The unit, defaults to 1, use 8192 to produce shorter but less accurate info */
integer total_length;       /*  The total length of the bytes written since    the last check point  */
integer options;            /* unsigned options */
integer lastv;              /* compression trick if
|synctex_options&4|>0.  */
integer form_depth;        /*  pdf forms are an example of nested sheets */
struct _flags {
unsigned int option_read:1; /*  Command line option read (in case of problem or at the end) */
unsigned int content_ready:1; /*  Command line option read (in case of problem or at the end) */
unsigned int off:1;         /*  Definitely turn off synctex, corresponds to cli option -synctex=0 */
unsigned int no_gz:1;       /*  Whether zlib is used or not */
unsigned int not_void:1;    /*  Whether it really contains synchronization material */
unsigned int warn:1;        /*  One shot warning flag */
unsigned int quoted:1;      /*  Whether the input file name was quoted by tex or not, for example "\"my input file.tex\"", unused by XeTeX */
unsigned int output_p:1;    /*  Whether the output_directory is used */
unsigned int reserved:SYNCTEX_BITS_PER_BYTE*sizeof(int)-8; /* Align */
} flags;
} synctex_ctxt = {


To also record the column in the input file, you'd have to add another field to the data structure and adapt all the recorder functions to write this entry to the SyncTeX file. The next step would be to adapt the SyncTeX parser to pick up the column from the SyncTeX file. Then you'd have to edit the source code of your editor to make it aware that inverse search now also maps to the column. The hardest part is, that all of the above has to be done in a backwards compatible way, such that your editor can still read SyncTeX files from older version and that SyncTeX files written with the newer version can still be read by old parsers.

So bottomlined, it is theoretically possible to implement your idea but finding someone to practically do this is probably NP-hard.

• Henri you are correct there is lots under the hood, however this comment is mainly for AbstractDissonance benefit the basic io interface has the following command structure --- Synctex view -i line:column:[page_hint:]input -o output [-d directory] [-x viewer-command] [-h before/offset:middle/after] --- where you can see basic usage is mainly the line and column number and in many viewers column is ignored as a problem so commonly simple viewers work just at line level. hence forward/inverse commands are just --- editor line filename – user170109 Nov 30 '18 at 0:48
• @KJO Really? I can't find any column information in the SyncTeX file. – Henri Menke Nov 30 '18 at 2:36
• Not a programmer, so I can only go by the file documentation and contents That would explain why %c column always returns an exceptionally high value, its possibly pre-set to the max but when parsing between editor and viewer not reduced down to a working value. – user170109 Nov 30 '18 at 8:24
• Just some pieces of the puzzle: The tag is an internal identifier for the box/record in question (do not mix it up with the form tag which is something different). Basically synctex supports column output (see the help of synctex edit) but only when written to the synctex file which (in my experience) does not contain it (i.e. the engines do not output it). If not present synctex will output -1 (pass that to the editor). – TeXnician Nov 30 '18 at 13:35
• @TeXnician ok now I see why -1 in some cases may be an exceedingly high value when unsigned. – user170109 Nov 30 '18 at 14:11

First of all, how does synctex work in general? Basically when editing code forward synchronization is performed by calling

synctex view -i line:column:[page_hint:]input


which is given line and column in the input file and tries to find the position in the output. So at least forward synchronization works the way you want it.

For backward synchronization the viewer calls

synctex edit -o page:x:y:file -x editor-command


which represents the page number, x and y coordinates of the mouse event and the file name. Within the editor-command you may actually use %{column} to pass the column to the editor. But be warned that column support is quite fragile.

According to the manpage that boxes and records (current, kern, glue, math) have column support, but that is optional. So basically the engine is allowed to output columns to the synctex file. In reality, however, you will notice that the link section only consists of the tag (an internal representation/id of the box or record) and the line number, so you would have to add engine support for that.

That's also the reason why your environment will not aid anything, because basically the "interpreter" of TeX, the engine does not know about it. That's the end you would have to work on, keeping in mind that some editors might not support column input either.

Referring to Is it possible to redefine the white space command?,

\documentclass{article}
\newcommand{\AND}{\par}
\obeyspaces\obeylines
\begingroup\lccode\~=\ \lowercase{\endgroup\let~}\AND
\begingroup\lccode\~=\^^M\lowercase{\endgroup\let~}\AND
\begin{document}
Blah blah blah. This is a sentence blah! Blah Blah!

\bigskip

Line1:
Blah blah blah. This is a sentence blah! Blah Blah!
Line2:
\end{document}