Breaking the WAL

john_strinlai

related: Tailscale Traces Database Corruption to 16y/o SQLite WAL-Reset Bug (tailscale.com) https://news.ycombinator.com/item?id=49272832

wwilson

We wanted to publish this ASAP since people are talking about this bug today. In a follow-up, we will show how our automated causality analysis could have saved Tailscale and SQLite from 6 months of root cause analysis. Stay tuned!

carlsverre

Hi! I'm the author and the person who was on a road trip when I decided to do this experiment back in July. I'm super excited to finally get the blog post out, and even better, alongside a post describing the root cause process that went into the project. You can read about Tailscale's story here: https://tailscale.com/blog/sqlite-wal-reset-bug I'm only sad that I didn't put SQLite under test earlier in the year, or I would have found this issue right away. If you look at the workload[1], you can see how simple it is. Exactly the same kind of workload we write every day to help our customers find bugs. If you have any questions about our process or how debugging with Antithesis works, please let me know! Thanks for reading! [1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...

minimaltom

I went clicking through to see if I could find the prompt they fed the AI to locate the issue / write the test suite. I couldn't find it, so its unclear if the prompt was completely "make a test suite" or was lead towards finding it in the first place, which wouldn't be a fair test. The closest I mention of the prompt I could find was: > Then I asked it to write a simple workload which exercised the WAL insert and checkpoint code. Notably, this is a completely generic workload. With a skeptical lens, unclear.

uhohherewegoaga

as a long time lurker who usually enjoys antithesis posts, this was a pretty existentially sad read. we all know that company blog posts here always tend to be ads at their core, but historically afaicr antithesis generally toes the line well of keeping the "thing we sell is really good" as a side dish to the "i (author) did very interesting deep thinking and/or hard work to accomplish something cool" entree. this one falls very flat on that front imo. the "antithesis finds rare bugs" is the overwhelming majority of the actual content here, where as the usual meat of the blog, the hard work, the deep thinking, really distills down to "typed this into claude mobile, typed that into claude mobile". yes, its a statement to the power of the technologies of today that's all it takes to find this bug, but man it's a sad signal for those of us looking for compelling technical reads instead of b2b sales pitches.

archseer

Related: finding the same bug with TLA+ https://news.ycombinator.com/item?id=48730953

LoganDark

I love hearing about this product, I hope one day I'll get to use anything like it.

nemothekid

I understand the author feels he missed an opportunity to catch a bug in Sqlite - but this still feels like catching something in hindsight. Just knowing the bug the is in Sqlite's concurrency engine is a huge win. But reading over tailscale's blogpost, I'm not sure how I would connect the dots to "this issue I'm seeing in production" to "load this up in antithesis". It seemed half the battle was figuring out if the bug was in Sqlite, Tailscale, or Linux, and in that situation if you gave Antithesis all the context, would it reliably find the bug?