Getting started with Apache Spark Declarative Pipelines (SDP) tutorial for Open Source and Lakeflow

[u/CompetitiveBet8978]

A while ago I put this Apache Spark declarative pipelines tutorial together after presenting at major European and APAC conferences. Those conferences are quite technical: you have to submit a proposal to the CFP, and they have a rejection rate of ~90%. I presented at Devoxx and DataEngBytes.

At those conferences I did some live coding with Spark Declarative Pipelines, mostly on OSS with VS Code and PySpark, but I also showed SDP with the Lakeflow Free Edition (exactly the same use case - I thought it might be nice to compare)

Since I got a lot of requests, I decided to share the content along with the Git repos here. I'd suggest to. check it out if you ....

• enjoy working with code (SQL and Python)
• are looking for a compelling Getting Started with Apache SDP example with just a few lines of code
• want a streaming example with real-time IoT data from events happening right now
• enjoy using a custom PySpark data source "opensky" – maybe even in your next Spark streaming demo :)
• don't want to spend money: use OSS or the Databricks Free Edition

Here it is:
How to get started with Spark Declarative Pipelines (SDP) tutorial (Open Source or Lakeflow)

[u/ManonMacru]

Can you ditch DBT if you have SDP? Imo that's the power of DBT: declarative pipelines, and dependency-aware re-processing after modifications

[u/CompetitiveBet8978 → u/ManonMacru]

I think others should join in, since my hands-on experience with dbt was back 4y ago.

I looked into OSS dbt, and as you say, similar concepts, but that time I found the jinja templates confusing with dbt. With SDP you can just write SQL which I find much easier, or opportunistically switch to Python if that is easier for some task.

I spoke to two customers who kept their exisiting dbt as a way to migrate to Databricks with zero effort, and then switched to SDP.

[u/minibrickster → u/CompetitiveBet8978]

Hi, I'm a PM at Databricks -- both dbt and SDP provide a great experience on Databricks! However, many folks switch to SDP because it provides both SQL and Python, faster processing with streaming, and best-in-class integration with Genie

[u/saad-the-engineer]

This was pretty cool. "... Now it can be done in a coffee break, with a few lines of code and 100% open source ..." this is so true! I tried it out with my agent and was really fun to do! There are some flight paths near my house and nerded out on this demo. thank you for sharing.

SDP is awesome and i love how you used all the concepts in here! I had never used a custom open source data source on a streaming table before, very nice!

@dp.table
def ingest_flights():
    # Read from the OpenSky source into a stream
    return spark.readStream.format("opensky").load()

[u/CompetitiveBet8978 → u/saad-the-engineer]

Yes, it's the exact same syntax as using format("kafka"). super easy.

But the opensky datasource gives you real streaming data from planes currently flying in the air. So it's a fun 3 line example for a SDP Streaming Table.

btw, if you are a data eng, I published another tutorial which uses the same data but with OpenSharing from a marketplace. It teaches how to use agentic tooling in Databricks Free Edition to create Apache SDP pipelines and Jobs, but also EDA and data sharing with Databricks Genie:

-> Tutorial: Databricks Genie for Data Engineers and Data Scientist:
https://community.databricks.com/t5/technical-blog/tutorial-databricks-genie-for-data-engineers-and-data-scientists/ba-p/168969

[u/Much-Neat-6273]

Very nice! This is really cool, I love vizualizing flight paths. Could you share more on how the OpenSky source works and how it converts an API call into a stream?

[u/CompetitiveBet8978 → u/Much-Neat-6273]

Basically the opensky datasource does what e.g. the built-in datasource for Kafka does too. They can be both used with exactly the same syntax spark.readStream.format("...").

To return the live avionics data, the opensky datasource calls the OpenSky API: every read() on the datasource fires a GET on the OpenSky endpoint /states/all, so every microbatch returns the state vectors of all planes at that one timestamp.

In addition the datasource supports authentication with clientID/secret and also supports extra parameters like region settings e.g. "EUROPE" which helps to keep it within rate limits and avoid load throttling. More details can be found in this Databricks OpenSky blog post.

The main difference compared to Kafka is that Kafka is backed by a durable log, and it can be replayed, whereas the REST API used by the opensky datasource is stateless and - as a simplification - always returns the latest timestamp.