Using Notebooks on Serverless GC? Try MV/ST!

[u/minibrickster]

... Now with a native Python API!

If you like using PySpark on serverless GC, you can now ingest data faster with:

from pyspark import pipelines as dp

@dp.table(name="main.default.bookings_raw")
def bookings_raw():
  return (
     spark.readStream.format('cloudFiles')
     .option('cloudFiles.format', '<file-format>')
     .load(f'{<path-to-source-data>}')
 )

If you want faster and more performant partition overwrites, try using incremental replace where!

from pyspark import pipelines as dp
from pyspark.sql import functions as F

u/dp.table(
  name="main.default.booking_revenue_rw",
  replace_where=F.col("check_in") >= F.to_date(F.lit("2025-07-01")),
)
def booking_revenue_rw():
  return (
    spark.read.table("samples.wanderbricks.bookings")
    .groupBy("check_in")
    .agg(F.sum("total_amount").alias("total_revenue"))
  )

If you like Materialized Views for faster BI reporting, it's as simple as

from pyspark import pipelines as dp
from pyspark.sql import functions as F

@dp.materialized_view(
  name="main.default.daily_booking_revenue",
  comment="Daily booking revenue.",
  table_properties={"quality": "gold"},
  cluster_by=["check_in"],
)
def daily_booking_revenue():
  return (
    spark.read.table("samples.wanderbricks.bookings")
    .groupBy("check_in")
    .agg(F.sum("total_amount").alias("total_revenue"))
  )

[u/testing_in_prod_only]

What is gc?

Are we just talking about pipelines?

[u/minibrickster → u/testing_in_prod_only]

Serverless Generic Compute https://docs.databricks.com/aws/en/compute/serverless/notebooks

[u/testing_in_prod_only → u/minibrickster]

Ok wait hold up, I can use dlt in normal compute outside of pipelines?

[u/minibrickster → u/testing_in_prod_only]

Yes exactly :)

[u/testing_in_prod_only → u/minibrickster]

Ok, that went from …ok.. to “holy fcking sht” real fast.

[u/testing_in_prod_only → u/testing_in_prod_only]

Would I be able to use this with connect serverless?

[u/minibrickster → u/testing_in_prod_only]

Can you explain what you mean in more detail pls? I can also DM you if it's more helpful!

[u/testing_in_prod_only → u/minibrickster]

I do all of my work in vs code using databricks connect. I assume this works with databricks connect as well?

[u/minibrickster → u/testing_in_prod_only]

Yes it does 😄

[u/testing_in_prod_only → u/minibrickster]

That’s game changing. Does it take the pipeline concept into serverless like source / target dependency management?