Show HN: Pivot – fast analytics on Iceberg, 2x+ faster than ClickHouse / DuckDB

dkgs

Hi HN! This is Gilad, one of the creators of Pivot We built an open-source analytics database that allows companies to run real-time analytics workloads on top of open data formats (Iceberg, Parquet), instead of needing to store data in a closed-format datastore (e.g., ClickHouse, Druid…) for real-time analytics use cases. We've built the engine from scratch in Rust and use Arrow for in-memory data processing, alongside a thread-per-core execution model to fully control the scheduling of CPU work. To achieve the speed we wanted, we had to build a custom memory allocator, tinker with the way JOIN and GROUP BY algorithms work to fully utilize the memory bandwidth of modern CPUs, and, in general, do a lot of optimization work to ensure we achieve superior performance compared to other real-time analytics engines, while remaining committed to using only Parquet / open data formats! We're excited to share it with the community and hope others find it as useful as we have!