Show HN: Parseable, an open observability datalake, handles 100M time-series/min

yashdotrv

Hi HN, This is Yash, founding team at Parseable ( https://github.com/parseablehq ). We've built an open source observability data lake using Rust, that handles high-cardinality data at around 100M time series in production ( https://www.parseable.com/blog/how-parseable-handles-100-mil... ) Our architecture is built around columnar design, and we use Apache Arrow for in-memory columnar processing and Apache Parquet for durable columnar storage on S3-compatible object storage. In Parseable, every labels stay as columns in the data instead of becoming a large long-lived per-series index like many TSDBs. Also, one thing we’ve been thinking about a lot is how observability changes as agents become part of day-to-day engineering workflows. They're not just another service, they produce traces, tool calls, prompts, intermediate decisions, errors, costs, and sometimes sensitive business context. Observing them matters just as much as observing any other system. But it is equally important to decide where that telemetry data should reside. Our view is that teams should be able to keep these observability data close to them: in their own object storage, under their own retention, access, and compliance controls.

nwmcsween

Low level how does this compare to Victoriametrics, you can get the high cardinality with clickhouse and other columnar dbs but the tradeoff is more ram usage, slower queries, io, etc

usernametaken29

How’s this any different then Iceberg?

simonw

The open source version doesn't accept protobufs, but does accept JSON. I found this out because I set Codex the task of running this locally agains another of my apps and it worked around the limitation by running this proxy: https://gist.github.com/simonw/b0e61a0aa8e3f7d30f27ce2f747c9... ... but it turns out my stack can emit JSON just fine, so I switched to that instead. Here's me TIL write-up of getting Parseable running locally https://til.simonwillison.net/datasette/datasette-parseable-...

vivzkestrel

- comparison with timescaledb please?