Next-Level Benchmarking Overfitting – Thomas Neumann

qsort

The author is correct in pointing out that this is a continuation and amplification of an existing trend. I'm in the data space fighting the good fight, spreading the gospel of Just Using Postgres, and I'm seeing more and more benchmarks that look extremely dubious. Query performance is like choosing a data structure in ordinary programming: it depends almost entirely of what access pattern you expect at runtime; it's entirely pointless to go search for the world's fastest linked list if you needed a hash table. Compare with the situation in the modern data engineering landscape. Are you sure the solution measured by the benchmark is actually a solution to your problem, let alone a significant improvement over Just Using Postgres? Have you measured or are you going on marketing and vibes? Again, I'm sure solutions like this have a place, but at least right now it's not the same place as traditional engines.

perching_aix

> Which brings me back to my original blog post claim: Letting AI agents optimize for a given database benchmark makes the benchmark useless. Good! That means the benchmark was always prone to be gamed and misrepresented, and deserves to be obsoleted and ignored, or at the very least heavily disclaimed. It's frustrating that people are betraying common trust by making misleading claims and engaging in misleading arguments, but ultimately this will improve rigor and decision making. At least when it's a bounded topic like this, it's not the same as it is with everyday subjects, where misinformation ends in a death by a thousand cuts situation (for now anyways) and is generally endless.

lmeyerov

We have been building GFQL for open source vectorized CPU/GPU cypher on dataframes (think duckdb for graphs, no DB needed) using agentic assistance... and our thinking is this is a stepping stone to something even more aggressive Today, we embrace optimizing hotpath specializations for some patterns that would have been too expensive to maintain before. LLMs keep that cost bearable, and being able to do that for important users (like users of our analyst product Graphistry) is a superpower. Our scores on graph community benchmarks are beating the big guys - and importantly, on anything we care about. Where we really want that to go is runtime / loadtime specialization. Our customers would absolutely want their top queries autotuned. DB research is already heavily tilting here, akin to tracing compiler research a decade ago. I'm quite bullish on what is now possible.

augment_me

There has been a similar issue in the GPU-kernel dev community. It was solved by understanding that the value provided by the benchmarks depends on your use-case. It sounds like the author in the article is yelling at the benchmark from his own idea of what a DB search should be (reliable, general), and not from what the benchmark is measuring. If you just want the fastest kernel for a handful of shapes, let people make brittle but fast code. If you are interested in general broad solutions, benchmark 50 different entries, limit code size to disallow assembly branches for every possible entry. Put requirements like bitwise equivalence or other end-task limits in there. Have public & private datasets. I think it's more of a proof that we should spend more time thinking about what and why we want these things, and design benchmarks that reflect that instead of going of the old way of doing things.