final addition: when does <foo> cm = <bar> in
hold for TeX ?
passing mathematical details and barring mathematical oversight in my quick investigation it is necessary and sufficient that the rounding of <foo>
to an entire multiple f/65536
of 1/65536
gives an f
which is multiple of 127
.
Then and only then is there some <bar> in
which gives exactly the same TeX dimension. And the rounding of <bar>
to an integral multiple g/65536
of 1/65536
will be with g
a multiple of 50
. And vice versa.
Example: we seek such a dimension nearby 0.6in
. We need a multiple of 50
nearby 0.6*65536=39321.6
, hence 39300
or 39350
. The former is obtained via 0.59967in
(and decimals nearby) and the latter via 0.60043in
(and decimals nearby).
In fact, for TeX we have exactly
0.59967in = 1.52316cm = 2840211sp
0.60043in = 1.52510cm = 2843824sp
\number\dimexpr0.59967in\relax =\number\dimexpr1.52316cm \relax
\number\dimexpr0.60043in\relax =\number\dimexpr1.52510cm \relax
\bye

But, as is the main cause of the OP, 0.6in
has no exact equivalent using cm
unit. In terms of sp
units the possible N sp
which admit both a representation as <foo> in
and <bar> cm
are those with N = int(3613.5*k)
for some integer k
. Above, k
is respectively 786
and 787
.
update: explanation how TeX scans dimensions
It has been clarified below and in comments that the cause of the difference was in the fact that you used formerly in
as dimension units, and in your reconstruction attempt were using rather cm
. Further, it is not exactly the same to make the difference between two dimensions originally expressed in in
and to specify directly the result in in
's because 1in
is not an integral number of sp
, the smallest unit used by TeX.
In https://tex.stackexchange.com/a/231281/4686 I explained how TeX handles dimensions like abc.xyz... pt
. This is from §452 of tex.pdf
(texdoc tex.pdf). Turns out that part is reused also for abc.xyz... in
, hence I recall the result:
the fraction abc.xyz... is rounded (the rounding being away from zero) to an integral multiple of 1/65536
in this process, the algorithm may discard (without affecting the theoretical result) all except the first 17 digits after the decimal mark.
Let's say that we have the "unpacked" result in the form of a pair (n, f)
which represents at this stage the fraction (65536n+f)/65536
.
Now TeX takes into account the dimension unit. This is explained at §458 of tex.pdf
. The easy case is pt
(which was handled earlier at §453), TeX simply puts 65536*n+f
in the register. For the case of in
, we morally need to do (65536*n+f)*7227/100
. Of course the issue here is to not create overflows with integer arithmetic. The routine which does that is at §107. Let us write x=65536*n+f
. The operations (simplified here, no signs, no overflow intercept) are
t<- (x mod 32768)*7227
u<- (x div 32768)*7227 + (t div 32768)
v<- (u mod 100)*32768 + (t mod 32768)
RESULT<- 32768*(u div 100) + (v div 100)
clarification to be very precise TeX applies this routine not to the integer x=65536 n + f
, but to the (half-word) integer n
, and the routine above also returns the "remainder" v mod 100
, which is then appropriately combined with f
and the scale 7227/100
, so that almost up to the end data is maintained in shape (N, F)
representing N + F/65536
where the exact F
has been truncated to an integer; and at the very last step the attach_fraction
sub-routine puts y = 65536*N+F
into the register. But this final integer y
is exactly as if the procedure above had been applied to x = 65536 n +f
. Which would not be possible directly due to arithmetic overflows.
Following steps x = 32768 q + r
, t = 7227r
, u=7227q + int(7227r/32768) = 7727x/32768 - 7227r/32768 + int(7227r/32768)
which mean that u
(which is an integer) =7227x/32768 - d
with 0<= d < 1
. Hence u = int(7227x/32768)
. Writing u =100*(u div 100) + (u mod 100) = 100*(RESULT/32768 - (v div 100)/32768) + (v/ 32768 - (t mod 32768)/32768) = 100*RESULT/32768 + (v mod 100)/32768 - (t mod 32768)/32768
, we get RESULT = x*7227/100 - 32768*d/100 - (v mod 100)/100 + (t mod 32768)/100
. But the d
was t/32768 - int(t/32768)
= (t mod 32768)/32768
, in fact. Thus all simplifies to RESULT= x*7227/100 - (v mod 100)/100
. As RESULT
is an integer, and as x*7227
is an integer, this means that, exactly, RESULT
is the truncation of x*7227/100
to an integer.
We can sum up the whole thing into:
first round the decimal abc.xyz..
into an integral multiple x
of 1/65536
. In doing this we need keep only 17 fractional digits of the input.
multiply by conversion factor 72.27
and then truncate to an integral number of sp
units.
Ah, damn'd I wanted to illustrate the cm
thing. Well then the conversion factor from §458 is 7227/254
and all of the above with 100
replaced by 254
goes through.
Let's look thus at 1.52400970458984374999999999999cm
. First we need only 17 digits after decimal mark. Count digits and see we can simplify input to 1.52400970458984374
. Then we need to multiply by 65536
and round. Take your preferred engine. Some like things like Maple, etc.. others enjoy xint
.
$ rlwrap etex -jobname worksheet
This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016) (preloaded format=etex)
restricted \write18 enabled.
**xintexpr.sty
entering extended mode
(/usr/local/texlive/2016/texmf-dist/tex/generic/xint/xintexpr.sty
(/usr/local/texlive/2016/texmf-dist/tex/generic/xint/xintfrac.sty
(/usr/local/texlive/2016/texmf-dist/tex/generic/xint/xint.sty
(/usr/local/texlive/2016/texmf-dist/tex/generic/xint/xintcore.sty
(/usr/local/texlive/2016/texmf-dist/tex/generic/xint/xintkernel.sty))))
(/usr/local/texlive/2016/texmf-dist/tex/generic/xint/xinttools.sty))
*\def\x #1;{\message{\xinttheiexpr[40] #1\relax}}% (this rounds)
*\x 65536*1.52400970458984374;
99877.4999999999993446400000000000000000000000
*\x 65536*1.52400970458984375;
99877.5000000000000000000000000000000000000000
*
we are at a border case here, and the first case gives 99877
. Then we do the second step which is 99877*7227/254
:
*\x 99877*7227/254;
2841775.9015748031496062992125984251968503937008
which we must truncate: the result is 2841775sp
.
In the second case we get first via rounding 99878
, and then we must evaluate 99878*7227/254
:
*\x 99878*7227/254;
2841804.3543307086614173228346456692913385826772
hence the result is 2841804sp
.
Let's now take the case of the 0.6 in
specification. We repeat the above. First 0.6*65536=39321.6
is rounded to 39322
. Then 39322*7227/100 = 2841800.94
is truncated to 2841800sp
.
You will find all these values below.
For the record 1.4in
: 1.4*65536=91750.4
, rounded to 91750
then 91750*7227/100=6630772.5
is truncated to 6630772sp
.
this is not an answer but is too long for a comment
were you using cm
as dimension unit back then ?
edit I think you were using 0.6in
. See at bottom.
there is a (surprising to me currently because I have completely forgotten the details of TeX input process for dimensions at this stage) sensibility to using cm
as dimension unit.
{
\dimen2=1.52400970458984374999999999999cm
\number\dimen2
\dimen2=1.52400970458984375cm
\number\dimen2
\dimen2=1.52400970458984374cm
\number\dimen2
}
\input xintexpr.sty
\xinttheiexpr [50] 1.52400970458984374999999999999*72.27/2.54*65536\relax
\xinttheiexpr [50] 1.52400970458984375*72.27/2.54*65536\relax
\xinttheiexpr [50] 1.52400970458984374*72.27/2.54*65536\relax
\bye

Update:
{
\dimen2=1.52400970458984374999999999999cm
\number\dimen2
\dimen2=1.52400970458984375cm
\number\dimen2
\dimen2=1.52400970458984374cm
\number\dimen2
}
\input xintexpr.sty
\xinttheiexpr [50] 1.52400970458984374999999999999*72.27/2.54*65536\relax
\xinttheiexpr [50] 1.52400970458984375*72.27/2.54*65536\relax
\xinttheiexpr [50] 1.52400970458984374*72.27/2.54*65536\relax
\number\dimexpr 2.54cm\relax
\number\dimexpr 1in\relax
\number\dimexpr 254cm\relax
\number\dimexpr 100in\relax
\xinttheiexpr [50] 1.52400970458984374/2.54\relax
{
\dimen2=0.6in
\number\dimen2
}
\bye

% TeX value
{
\dimen2=0.6in
\number\dimen2
}
% exact value:
\xinttheiexpr [10] 0.6*72.27*65536\relax
\bye
