There was an error while loading. Please reload this page.
1 parent 3d1c7de commit bae1b94Copy full SHA for bae1b94
1 file changed
Doc/tutorial/floatingpoint.rst
@@ -173,24 +173,24 @@ and recalling that *J* has exactly 53 bits (is ``>= 2**52`` but ``< 2**53``),
173
the best value for *N* is 56::
174
175
>>> 2**52
176
- 4503599627370496L
+ 4503599627370496
177
>>> 2**53
178
- 9007199254740992L
+ 9007199254740992
179
>>> 2**56/10
180
- 7205759403792793L
+ 7205759403792794.0
181
182
That is, 56 is the only value for *N* that leaves *J* with exactly 53 bits. The
183
best possible value for *J* is then that quotient rounded::
184
185
>>> q, r = divmod(2**56, 10)
186
>>> r
187
- 6L
+ 6
188
189
Since the remainder is more than half of 10, the best approximation is obtained
190
by rounding up::
191
192
>>> q+1
193
- 7205759403792794L
+ 7205759403792794
194
195
Therefore the best possible approximation to 1/10 in 754 double precision is
196
that over 2\*\*56, or ::
@@ -211,7 +211,7 @@ If we multiply that fraction by 10\*\*30, we can see the (truncated) value of
211
its 30 most significant decimal digits::
212
213
>>> 7205759403792794 * 10**30 / 2**56
214
- 100000000000000005551115123125L
+ 100000000000000005551115123125
215
216
meaning that the exact number stored in the computer is approximately equal to
217
the decimal value 0.100000000000000005551115123125. Rounding that to 17
0 commit comments