Shredding JSON columns gave Logfire up to 1000x faster queries

sans_souse

Let's just revert back 30 years and make the entire www plain text. What could possibly go wrong?

sgarland

It is hilarious to me when companies act like they've made some huge discovery that's been well-known to anyone working in DBs for the past decade, if not more. JSON is slow as hell, both for serdes, and especially querying it. Use it sparingly, if at all. Normalization is not that hard, and it's trivial now with LLMs. Tell Claude you want your schema in BCNF, or 5NF if you're feeling froggy. Denormalize if and only if you have performance issues.

japgolly

> Rather than storing all attributes in a single JSON column, we automatically extract the most frequently accessed attributes into separate, strongly typed columns. I'm glad they made the discovery but this is DB 101

zX41ZdbW

This is quite an old technique. In ClickHouse, it appeared before it was named "shredding": https://github.com/ClickHouse/ClickHouse/issues/54864

MrThoughtful

Why not store everything in a single json column but dynamically generate index files, so finding rows is fast? Then you never have to change the data. Only the index files.