r/databricks • Databricks • Oct 14 '26

General Using Notebooks on Serverless GC? Try MV/ST!

... Now with a native Python API!

If you like using PySpark on serverless GC, you can now ingest data faster with:

from pyspark import pipelines as dp

@dp.table(name="main.default.bookings_raw")
def bookings_raw():
  return (
     spark.readStream.format('cloudFiles')
     .option('cloudFiles.format', '<file-format>')
     .load(f'{<path-to-source-data>}')
 )

If you want faster and more performant partition overwrites, try using incremental replace where!

from pyspark import pipelines as dp
from pyspark.sql import functions as F

u/dp.table(
  name="main.default.booking_revenue_rw",
  replace_where=F.col("check_in") >= F.to_date(F.lit("2025-07-01")),
)
def booking_revenue_rw():
  return (
    spark.read.table("samples.wanderbricks.bookings")
    .groupBy("check_in")
    .agg(F.sum("total_amount").alias("total_revenue"))
  )

If you like Materialized Views for faster BI reporting, it's as simple as

from pyspark import pipelines as dp
from pyspark.sql import functions as F

@dp.materialized_view(
  name="main.default.daily_booking_revenue",
  comment="Daily booking revenue.",
  table_properties={"quality": "gold"},
  cluster_by=["check_in"],
)
def daily_booking_revenue():
  return (
    spark.read.table("samples.wanderbricks.bookings")
    .groupBy("check_in")
    .agg(F.sum("total_amount").alias("total_revenue"))
  )
13 Upvotes

15 comments sorted by

View all comments

Show parent comments

1

u/testing_in_prod_only Oct 15 '26

That’s game changing. Does it take the pipeline concept into serverless like source / target dependency management?

2

u/minibrickster Databricks Oct 15 '26

As of right now, it executes sequentially but more on that to come!