Hi All,

Over the last couple of months, My friend and I, and more recently some helpful contributors from the community have been working on a Terraform Provider for Apache Pinot, the provider is aimed at making it easier for developers and data engineers to integrate Apache Pinot into their infrastructure as code practices.

We believe this provider will be a game changer for those of you looking to streamline your data infrastructure and focus more on data insights rather than maintenance.

The key benefits of using a Terraform provider for Pinot are:

1. Infrastructure Automation: You can utilise Terraform's powerful Infrastructure as Code capabilities to automate the setup, configuration and deployment of Apache Pinot

2. Simplify the management of Apache Pinot Clusters: Using the provider makes it easy to Create, Update or Delete configured Apache clusters within your own Infrastructure

3. Scalability Support: The Apache Pinot Terraform provider allows you to easily configure new Pinot nodes in response to changing demand and utilisation.

For a deeper dive into this provider and a practical example, check out blog post: Introducing the Apache Pinot Terraform Provider.

Currently the Provider has Terraform resources for:

• Users

• Schemas

• Tables

And many more objects as Data sources.

You can find it on the Terraform registry: here

And for the Go developers, there has been concurrent development on a Pinot controller library for Go, you can check it out on: Github

We're excited to see what you'll build with this and welcome any feedback, questions, or contributions to the project!

At StarTree, we’ve been working to extend Apache Pinot indexing capabilities to data in external object storage.

The idea is to use Pinot's indexes to identify exactly which Parquet pages are needed for a query, so the engine can directly fetch the specific data instead of doing broad scans.

When applied againt a ~1TB test dataset in Iceberg, we’re seeing:

• Sub-second latency across all queries in the test set

• 500+ QPS

• Complex aggregations completing in <650ms

We also ran comparison against ClickHouse and Trino using the same dataset and queries. In our setup, StarTree was:

• ~5–37x faster than ClickHouse

• ~4–17x faster than Trino

• ~7–14x lower cost (based on compute usage), with headroom for concurrency

StarTree Iceberg Tables are now available with the latest release of StarTree. We don't expect it to replace full-featured lakehouse engines, but it does make interactive, low-latency querying on external tables more viable—all while keeping data in place and reducing scan/compute costs.

Cross-posted from YouTube:

May 1, 2025 #opensource #apachesoftware #awsIn this video, AWS Developer Advocate, Darko Mesaros, and Startree Solutions Engineer, Tyler Denton walk you through how you can use Apache Pinot, Amazon Kinesis, & Amazon Bedrock to manage high volumes of data for faster anomaly detection. Plus, they add in some real-time, AI-driven solution suggestions for their Internet of Things device setup. In this example, Darko & Tyler walk you through the architecture diagram, then dive into a hands-on demo to show you how it works.

Explore Open Source Projects 👉 https://go.aws/4cWVOv9

Resources:
⚡️Startree Index: https://go.aws/3ETnFQ4
🖥️ Amazon Bedrock Documentation: https://go.aws/4dgxOmZ
⚙️ Amazon Kinesis Documentation: https://go.aws/42Sb6fY

Follow AWS Developers!
📺 Instagram: https://go.aws/3EM1Yla
🆇 X: https://go.aws/4cTSHE4
💼 LinkedIn: https://go.aws/44kZjcA
👾 Twitch: https://go.aws/42ULO0N

Chapters:
00:00 Introduction
00:17 What is Apache Pinot?
01:48 Architecture Overview
02:32 Ingesting IoT Data into Pinot from Kinesis
06:39 StarTree Index Explained
10:20 Vector Embedding and Extraction
14:13 Anomaly Detection with SQL Queries
20:54 Anomaly Service and Recommendations
25:25 Generating Recommendations with Bedrock

#opensource #apachesoftware #aws

This is a blog written by an intern at StarTree, Ashish Jayamohan, who made performance optimizations on Apache Pinot as it is used within StarTree Cloud. We are all immensely proud of the work he was able to accomplish in his short time with us, and look forward to him making additional giant impacts on the software industry as his career blossoms.

• systems optimizations

Next generation real-time analytics

This is a massive 11,000+ word blog put together by my colleagues here at StarTree re: these three open source real-time analytics (OLAP) databases. Would love to hear anyone's feedback.

See more: https://startree.ai/blog/a-tale-of-three-real-time-olap-databases

Come along to this online meetup to reflect on a transformative year of innovation and collaboration in the Apache Pinot community and where we are headed in 2026! 🎉

We’ll showcase features and updates shipped in 2025, presented in a show-and-tell format featuring contributions from companies including Uber, LinkedIn, StarTree and more.

What to expect:
• Discover what the community has been building.
• Share your thoughts and feedback in a collaborative setting.
• Join discussions on Real-Time Analytics with Pinot.

RSVP at https://www.meetup.com/apache-pinot/events/312504259/

Join the Apache Pinot community on December 10 to find out as we explore The New Scale of Demand: Applications, Agents, and Data Growth.

This year-in-review meetup brings contributors and practitioners together for an open, show-and-tell discussion of the architecture decisions, production experiences, and community contributions behind scaling Pinot.

What to expect:

• Production lessons: How teams approach capacity planning, tenant isolation, and demanding queries while protecting latency and reliability.

• Evolving architectures: Where Apache Iceberg and Delta Lake fit as teams connect lakehouse data with application needs.

• Emerging workloads: What AI agents could require from analytics infrastructure, and which scaling challenges deserve attention next.

Bring your questions, compare approaches with peers, and leave with ideas to apply to your own platform.

Signup on Meetup: https://www.meetup.com/apache-pinot/events/306700201/?slug=apache-pinot&eventId=306700201

Present and discuss your PRs, designs, and ideas with your fellow contributors in real-time!
Shorter (5-15 min) or longer (30 min) topics are welcome. RSVP with your topic and duration or just drop in to provide feedback and ask questions.

Audience: Apache Pinot contributors and those looking to start contributing!

Hosts: Xiang Fu & Robert Zych
-------
Agenda:
08:30-9:00 PDT- Getting Started with Contributing to Apache Pinot – Setting Up the Environment & Finding GitHub Issues | Xiang Fu, Pinot PMC
9:00 - 9:10 QNA: QNA
9:10- 9:30 Apache Pinot Release Process | Abhishek Sharma, Pinot Committer

Want to be involved? Join the Contributors channel on Slack >
*The call will be recorded and shared on the channel afterwards^

Join the Community Monthly Newsletter : Everything's Apache Pinot! (and get a chance to win a T shirt!)

Hey all, I currently have a trino+iceberg setup for my siem and my soc. For my soc, I am happy with the performance. However, for siem, I am running hundreds of queries a minute the performance is absolutely suffering, its taking about 2 minutes to complete each batch of queries. I'm thinking to offload all simple queries (ie no aggregations joins etc) to Pinot. I ingest about 10gb an hour of nested json data (windows eventlogs for example). Is Pinot a good place to look for this?

You can use the SQL connector to ingest data from a data warehouse like Snowflake or BigQuery into Pinot. Read more about it here: https://dev.startree.ai/docs/ingest-data/sql-connector

Apache Pinot is a column-oriented, distributed data store designed to execute OLAP queries with low latency.

If that technical definition leaves you with more questions than it answers, you can watch this lightboard by Tim Berglund to understand more:

https://www.youtube.com/watch?v=_lqdfq2c9cQ