How much precision can you squeeze out of a table?
nomemory
48 points
6 comments
March 26, 2026
Related Discussions
Found 5 related stories in 40.1ms across 3,471 title embeddings via pgvector HNSW
- Reports of code's death are greatly exaggerated stevekrouse · 341 pts · March 22, 2026 · 39% similar
- Supertoast tables abelanger · 51 pts · March 06, 2026 · 38% similar
- Lego's 0.002mm specification and its implications for manufacturing (2025) scrlk · 362 pts · March 11, 2026 · 37% similar
- NumKong: 2'000 Mixed Precision Kernels for All ashvardanian · 39 pts · March 20, 2026 · 37% similar
- How many branches can your CPU predict? ibobev · 122 pts · March 19, 2026 · 36% similar
Discussion Highlights (2 comments)
PaulHoule
When I was in high school I thought book this was so much fun http://182.160.97.198:8080/xmlui/bitstream/handle/123456789/... which is all about the kind of numerical analysis you would do by hand and introduces a lot of really cool math like the calculus of differences https://en.wikipedia.org/wiki/Finite_difference
gus_massa
Nitpicking: > You’re never going to get error less than 10E−15 since that’s the error in the tabulated values, If you have like 100 (or 400) values in the table, you can squeeze one more digit. In the simple case, imagine you have the constant pi, with 15 digits, repeated 100 times. If the rounding was done in a random direction like floor(pi * 1E15 + random() - 1/2 ) / 1E15 then you can use statistic to get an average with an error that is like K/sqrt(N) where K is a constant and N is the number of samples. If you were paying attention in you statistic clases, you probably know the value of K, but it's not my case. Anyway, just pick N=100 or N=400 or something like that and you get another digit for "free". Nobody builds a table for a constant and uses a uniform offset for rounding. Anyway, with some smooth function that has no special values in the points where it's sample, the exact value of the cut decimal is quite "random" and you get a similar randomization.