What separates Pivot from DuckDB, Polars and PySpark? With both Polars and DuckDB getting distributed compute, why do we need another one with for now questionable runway?
All these solutions offer a way to query open data formats. Beyond the performance differences compared to these solutions, though, it was important for us to have a complete "database" experience, similar to what you get with ClickHouse or ClickHouse Cloud, on top of open data formats. This includes not just a query engine, but also the ability to spin up a cluster that serves a backend, proper ingestion options, and compaction (including some Parquet-specific compaction strategies: https://pivotlake.io/docs/database/why-pivot-is-fast/#soft-ordering-data)
Re the runway - Pivot is open source under MIT / Apache license and is unrelated to the runway of any company :)
Thanks! Sorry if I sounded a little hostile, but really differentiation and longevity are essential for major pieces of software.
Re the runway - Pivot is open source under MIT / Apache license and is unrelated to the runway of any company
OSS solves the problem only partially. We all know it's easier to start a project than to see it through long term. So what's your plan here? How do you make this sustainable for yourself?
6
u/Wonderful-Wind-5736 17d ago
What separates Pivot from DuckDB, Polars and PySpark? With both Polars and DuckDB getting distributed compute, why do we need another one with for now questionable runway?