SBD: We Have a Filesystem at Home

dangoodmanUT

I really like fly.io posts, they're usually written in such a way that feels so human. I used to read all of them, because the writing was interesting, and secondarily sometimes the topic was. Except seeing sentences like this: > The difference is that a Sprite now stops and says so instead of carrying on with a disk it can’t vouch for Makes it feel distinctly LLM written. Maybe I'm being RL'd to identify LLM authorship, maybe the author was RL'd to write like that by reading so many Claude-written works, but I'd like to see the humanity of fly.io blog posts come back.

themgt

A story which may or may not be based on lived experience: one day a Sprite stops booting. JuiceFS is asking object storage for a chunk and getting a 404. We wanted to go fast, so we ran JuiceFS in writeback mode. It records a chunk in its metadata first and uploads the bytes after. If the machine is killed at any time between those two steps, the metadata points at a chunk that doesn’t exist. So we built a harness that drives fleets of Sprites through write, checkpoint, sleep, wake and restore, over and over, verifying every byte that comes back. We pointed it at 400 JuiceFS Sprites for about 5,000 cycles. A nonzero number of them permanently lost the ability to checkpoint. Well … fuck. Enter: the Sprites Block Device (SBD). It takes a bucket and presents it to the Linux kernel as a block device, over NBD. The Sprite formats that device as ext4 and mounts it, the same way you’d mount a disk on any Linux box. Duh. This blog has has a strange vibe like "the last BRAWNDO we made in a bathtub may or may not have killed some customers lmao, try NEW BRAWNDO 2.0 TODAY!" that I didn't realize could work for corporate messaging around an in-house CoW cloud filesystem.

bennettnate5

> I have unkind things to say about people who make their own pasta You lost me on the first sentence; my significant other's homemade ravioli is unparalleled. Homemade pasta is a delight to make and tastes so much better than $1 cardboard box pasta

aftbit

I'll be honest, I don't understand what Sprites are, or why they need their own kind of block device. All of the pain seems to come back to trying to map a linux filesystem running on a block device into an object store. They could just ... not do that? Here's why they do: https://fly.io/blog/design-and-implementation/ But it feels like maybe they made the wrong tradeoff here.

xyzzy_plugh

So it's like Archil or ZeroFS but more proprietary and less useful. Got it. Look, we get it: anyone can write a filesystem now. It doesn't mean you should .

stego-tech

While I love the post, I just kinda have to lol that it all ends up back where we started with ext4 atop a block device. The more things change, the more they stay the same. Now look, I’m not about to try and stop folks from finding new ways of solving problems, especially in infrastructure. Far from it, I love seeing what you beautiful diamonds come up with that works for you. They make for amazing reads! Except 99.999% of infrastructure needs to be boring, bland, basic, and bulletproof . Most Infra folks will take the minor inefficiencies associated with object storage because we’re not a unicorn startup where solving such a problem could save us millions of dollars a year in write calls to our hypercloud provider; we’re far too busy trying to get our internal customers to stop overscheduling VMs or implement the latest executive shiny project. We need our underlying abstractions - hypervisors, storage arrays, network cores - to work , without fuss and with minimal maintenance. So while it’s super neat that Fly.io has shared this, and they absolutely have a relevant and beneficial use case for reinventing the proverbial wheel, it’s important to note they are the edge case , not the norm.

ublkcheersquad

I wonder if the folks at fly.io are aware of ublk ( https://docs.kernel.org/block/ublk.html ). It seems a perfect fit for this use case - I imagine their scale would benefit from a uring-based in-kernel driver that supports zero-copy thru the dataplane.