Distributed, high-performance blob storage on SQLite
ghostfoxgod
277 points
71 comments
November 19, 2026
Related Discussions
Found 5 related stories in 135.8ms across 13,334 title embeddings via pgvector HNSW
- SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers ankitg12 · 238 pts · July 29, 2026 · 57% similar
- Twelve years of distributed SQLite otoolep · 109 pts · October 24, 2026 · 57% similar
- Lobste.rs is now running on SQLite abetusk · 29 pts · July 13, 2026 · 54% similar
- Lobste.rs is now running on SQLite ahamez · 12 pts · July 14, 2026 · 53% similar
- The Fastest Object Storage Client in the World solatic · 13 pts · November 09, 2026 · 51% similar
Discussion Highlights (20 comments)
feverzsj
Sounds like you should use a database-per-user design.
CyrusGrissom
I couldn't find the word "replication" in the post or on the GitHub page. So, what about fault tolerance?
bob1029
> JSON blobs An alternative to faster blob storage is to represent the blob's schema in SQL directly. If most blob updates are incremental you can eliminate a significant amount of storage IO and garbage collection this way. One SQLite database per blob type with a metadata SQLite db that tracks each. We could leverage powerful AI reasoning models to produce SQLite schemas and virtual table adapters for each encountered JSON schema type.
zodwick
all roads lead to sqlite
robinduckett
What about using, ahem, files?
_s_a_m_
so it's just sharding?
matja
What is the consistency model? What happens if a node is down unexpectedly?
Kalanos
A limitation that i ran into with sqlite was not being able to interact with it over the network e.g. NFS. Does this solve that problem?
Kalanos
Is the db still a single, portable file?
martinjc
Why not rqlite?
murkt
Sounds like it solves their particular problem, but hardly can be used by outsiders in the current state. Bad fault tolerance, no words on replication or backup. Offline rebalance... meh. I am not sure how I feel about their choice of Redis protocol.
anacrolix
I spent years trying to squeeze this use case out of sqlite. All roads lead to tears, it is not made for this use case. I wrote https://github.com/anacrolix/possum for this exact use case. It has a few rough edges but it absolutely hauls ass for caching on disk. Incidentally it uses sqlite for transaction/TTL metadata which is more suited to sqlite but it's still the bottleneck. sqlite transaction commit overhead is too high and uncompetitive without batching but it's a design flaw specific to being multiprocess.
ratherlongname
So many comments about things already addressed in the article sigh
Blobtits
Reads like this is be a perfect fit for ClickHouse. You should do async inserts, remove the 20 or so _log options ClickHouse enables for some insane defaults and batch them in Redis if needed (probablynot). Then you have automatic compression, free ttl, no write concurrency.
gkamal
This looks like a classic case of nih/reinventing wheel.
lukax
OpenStack Swift uses SQLite for storing metadata. Each account has one SQLite db. Each container also initially had one SQLite db but in 2018 they added support for sharding so if a container has more than 1 million objects, the container is split into multiple SQLite databases. The performance is not ideal. Ceph scales much better.
redwood
To confirm this is being stored on devices rather than on a server side environment?
articulatepang
I wonder if they tried storing diffs instead, and compacting them once in a while. These are user preferences for UI setup, and they have a lot of write traffic. But probably each user changes their preferences only by a little bit each time. Their 1MB writes would go down to just the small diffs. They could store the diffs and transparently apply them before sending to the client. After some timeout or maybe after a certain number of diffs have accumulated, squash them all and write back. They’d still need a server and custom code to handle this but it would probably reduce the database load and SSD writes by a lot. And let them just use something off-the-shelf for storage.
sgarland
> What we learned along the way… Deleting rows doesn’t shrink SQLite files This is true of every RDBMS I am aware of. It would be horribly inefficient to constantly run page merges in order to be able to free some of them. Whenever I read things on posts that are a. discussed in documentation b. should be apparent from first principles, I immediately lose faith in the product.
nodesocket
If only rethinkdb hadn’t nose dived. It supported sharding, health checks, clustering with failover. Write performance was quite good as I recall.