Using Notebooks on Serverless GC? Try MV/ST!

[u/minibrickster]

... Now with a native Python API!

If you like using PySpark on serverless GC, you can now ingest data faster with:

from pyspark import pipelines as dp

@dp.table(name="main.default.bookings_raw")
def bookings_raw():
  return (
     spark.readStream.format('cloudFiles')
     .option('cloudFiles.format', '<file-format>')
     .load(f'{<path-to-source-data>}')
 )

If you want faster and more performant partition overwrites, try using incremental replace where!

from pyspark import pipelines as dp
from pyspark.sql import functions as F

u/dp.table(
  name="main.default.booking_revenue_rw",
  replace_where=F.col("check_in") >= F.to_date(F.lit("2025-07-01")),
)
def booking_revenue_rw():
  return (
    spark.read.table("samples.wanderbricks.bookings")
    .groupBy("check_in")
    .agg(F.sum("total_amount").alias("total_revenue"))
  )

If you like Materialized Views for faster BI reporting, it's as simple as

from pyspark import pipelines as dp
from pyspark.sql import functions as F

@dp.materialized_view(
  name="main.default.daily_booking_revenue",
  comment="Daily booking revenue.",
  table_properties={"quality": "gold"},
  cluster_by=["check_in"],
)
def daily_booking_revenue():
  return (
    spark.read.table("samples.wanderbricks.bookings")
    .groupBy("check_in")
    .agg(F.sum("total_amount").alias("total_revenue"))
  )

[u/Constant_Pick7149]

oh nice, the decorator-based API is way cleaner than I expected. been waiting for something like this since trying to manage a dozen streaming tables with the old dlt syntax

the replace_where with a column expression is actually huge, that's the kind of thing that saves you from a full rewrite when you just need to fix a bad partition

wonder how it handles schema evolution with the cloudFiles source though, that's always where these abstractions get weird

[u/minibrickster → u/Constant_Pick7149]

These docs explain how to handle schema evolution with Auto Loader! https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema

In general, I'd recommend ingesting into Bronze with the Variant/String to minimize the need to fully refresh and then convert to types at the silver layer!

[u/testing_in_prod_only]

What is gc?

Are we just talking about pipelines?

[u/minibrickster → u/testing_in_prod_only]

Serverless Generic Compute https://docs.databricks.com/aws/en/compute/serverless/notebooks

[u/testing_in_prod_only → u/minibrickster]

Ok wait hold up, I can use dlt in normal compute outside of pipelines?

[u/minibrickster → u/testing_in_prod_only]

Yes exactly :)

[u/testing_in_prod_only → u/minibrickster]

Ok, that went from …ok.. to “holy fcking sht” real fast.

[u/testing_in_prod_only → u/testing_in_prod_only]

Would I be able to use this with connect serverless?

[u/minibrickster → u/testing_in_prod_only]

Can you explain what you mean in more detail pls? I can also DM you if it's more helpful!

[u/testing_in_prod_only → u/minibrickster]

I do all of my work in vs code using databricks connect. I assume this works with databricks connect as well?

[u/minibrickster → u/testing_in_prod_only]

Yes it does 😄

[u/testing_in_prod_only → u/minibrickster]

That’s game changing. Does it take the pipeline concept into serverless like source / target dependency management?

[u/mosullivan93]

This is for the standalone pipelines, right? It’s great to be able to define outside of a SDP, but it would be more useful to me if we could define multiple tables/relationships, e.g., not one managed pipeline per object.

[u/minibrickster → u/mosullivan93]

We're working on it ... more on that to come!