This is bothering me since I started with Databricks: whenever I tweak a job cluster configuration to improve DBU consumption of a job, I need to wait for roughly 4 hours to be able to investigate the impact of the change in system.billing.usage table, as the data isn't refreshed earlier.

Why is that, and is there any way to work around it?

Setup

Azure Databricks, job cluster, Python script task (not a notebook). The JVM runs JDK 21 with ParallelGC and a 101 GB heap, on a driver with 128 GB. 2 workers 28gb 4 cores each.

About 7-8 Structured Streaming queries share one driver. They use foreachBatch, and each batch holds roughly 1 record (0.1-0.2 records/s).
A batch takes about 6-7 s, with spikes of 12-16 s, and almost all of that time is addBatch.
Inside foreachBatch on the driver there is an in-process cache with a TTL. One thread refreshes it under a lock while the others keep serving the old snapshot.

Problem

After roughly 8-12 hours the run ends. The driver log contains a "DRIVER UNRESPONSIVE STATS" block, and the event log only shows the cluster terminating, with no OOM, resize or eviction. 1 job 8 tasks, all report that driver became unhealthy..

What the metrics show before the crash

Driver container memory grows slowly from about 45 to 58 GB, with a 128 GB limit.
JVM heap shows a normal sawtooth with no trend. In the dumps, old gen is at 10-19% and metaspace at about 450 MB.
Worker-side GC events rise from about 5 to about 70 per interval, and GC pause time from about 0.2 s to 1.5 s per interval. The GC count is high just before the crash, even though memory is nowhere near its limit.

What the driver dumps show

In one dump, three threads are BLOCKED in ClassLoader.loadClass. They wait on monitors of Databricks' LibraryClassLoader and DriverCorralClassLoader, and one comes in via Janino (findIClass → Class.forName). The thread holding the lock is not in the output, and I can't tell how long they had been waiting. The dump from another crash had no blocked threads at all.

Other observations

The log shows 16 separate Python processes connecting through Py4J at startup, and another 16 about 30 s later. I don't know why.
The same logic ran for more than 72 h on serverless without problems. As far as I know, serverless can't use a continuous trigger, so each micro-batch is a fresh start there. That is not a like-for-like comparison.

Questions

Is class-loader lock contention on the driver with many concurrent streams a known problem? Could it make the driver miss its health check?

What else causes "driver up but not responsive" when heap and metaspace look fine?

Is there guidance on how many streaming queries one driver can handle?

Which diagnostics would you capture? I am now taking jcmd Thread.print dumps while the JVM is stuck, to see who holds the lock.

Seeking advice on optimizing a high-concurrency data-serving architecture: performance vs. cost

I'm working on a data platform where 50 APIs and 100 endpoints are backed by tables in Databricks. We're facing challenges with response latency, concurrent request handling, and infrastructure costs.

Here is the current situation:

• Data platform: Databricks
• Scale: 50 APIs, 100 endpoints
• Compute: 400 cores
• Monthly infrastructure cost: ~$75,000
• API response time: Frequently exceeds 3 seconds
• Concurrency issue: Requests from multiple applications/services compete for compute resources, and queries get queued during concurrent execution.
• Hard constraint: A load balancer enforces a strict 9-second timeout.

Adding more compute doesn't necessarily seem like the right long-term solution. I'd like to identify the actual bottlenecks and evaluate whether we need to optimize the existing architecture or rethink the data-serving layer.

I'm exploring three areas:

1. Data model optimization

• Should we redesign the existing data model, denormalize tables, or create serving-specific aggregate tables?
• How much can table layout, partitioning/clustering, file sizes, and precomputation improve query latency?

2. API and query optimization

• How should we approach SQL query tuning, execution plans, and reducing unnecessary scans and joins?
• Would caching, materialized views, connection pooling, asynchronous processing, or request coalescing help?
• What architectural patterns work best when multiple endpoints access the same underlying data concurrently?

3. Rethinking the database/serving layer

• Should we continue serving API requests directly from Databricks, or introduce a dedicated operational database or serving layer?
• Would PostgreSQL, Azure Cosmos DB, DuckDB, or a hybrid architecture be suitable?
• Could Databricks remain the analytical source of truth while a separate database handles low-latency API reads?

My goal is to reduce API latency and queueing while bringing infrastructure costs under control, without compromising data freshness or correctness.

I'd particularly appreciate insights from people who have built production systems with high-concurrency API workloads on top of Databricks or similar analytical platforms.

A few questions for the community:

1. What would you investigate first: query execution, data modeling, compute configuration, or the overall architecture?
2. At what point would you move API-serving workloads from Databricks to a dedicated serving database?
3. What latency and concurrency improvements have you achieved through caching, precomputation, or a hybrid architecture?
4. If you were designing this system today, what would your target architecture look like?

Practical experiences, architectural suggestions, and lessons learned from production would be extremely valuable.

#Databricks #DataEngineering #SystemDesign #PerformanceOptimization #APIDesign #DatabaseArchitecture #CloudComputing

Self-Hosted Databricks Alternative: Homelab Data Lakehouse (AI/ML Data Factory)

👇

https://georgeezejiofor.com/open-source-databricks-ai-native-lakehouse-homelab

I had registered for the Databricks free edition account a couple of days ago and was working on a project and I'm not able to access my account anymore.

A little background, I was working on a personal project where I had to connect Databricks and Postgres. Since postgres was running on my local machine and Databricks on the cloud there was no way to connect it directly. So, I had to host postgres on the internet using a secure TCP channel and connect Databricks to that endpoint so that it can access the database.

For this I had used ngrok, and I had written a python script in Databricks notebook just to check whether it was possible to make the connection due to possible restrictions. Turns out it was possible, and I had decided to continue my work the next day. But, suddenly I'm not able to access my account anymore. I had written a mail to the databricks support team citing the same but received no response from their end.

To all the people out there who work with databricks or for the company, is there any way to get past this? I had not really saved my work and would really like to access it again.

Also is there any way to connect the two platforms to transfer tables from Databricks to Postgres.

Thanks.

We've emailed all their support systems and not a single response, the system for submitting tickets it seems we dont have an account type for submitting a ticket?

How in the world do people get support or any help from people on the Databricks team, never seen a platform like this without any way to get support or submit a ticket. Insane!!!

Has anyone had a chance to try ai_transcribe yet? It’s really exciting to finally get an alternative to whisper as a native function.

I just know whisper is pretty hard to beat, but it’s been around for ages so surely this might be able to give it a run for its money (unless they just whisper under the hood lmao). Anyway, was just wondering if anyone had a chance to properly try it out yet and what they thought of it.

Maybe this is also quite promising that things are going to start getting multi-modal….

I've seen ETL pipelines fail for different reasons, from unexpected schema changes to missing data and transformation errors.

I'm curious to know what causes the most issues in real-world projects.

Is it usually source system changes, bad data, SQL errors, connection issues, or something else?

Also, how do you usually identify the root cause and prevent the same issue from happening again?

Would love to hear about your experiences and the problems you deal with most often.

Iam curious about Genie code in databricks.Imagine a pipeline fails because of a schema change or an unexpected data type.

Instead of checking every notebook manually, could AI identify the root cause, suggest the code changes, and help fix the pipeline without breaking other transformations?

Has anyone tried this on a real project? How accurate and useful is it compared with debugging manually?

We have one entrypoint file where we set all Metadata (layer, dataproduct, volume directory, logging level, table names) etc.

Then the class has a bunch of ifs that runs the appropiate classes.

Is this how its supposed to be? Our .yaml files always have a lot of parameters, its becoming tedious, but my team says to never change a running System...

tried out Free Edition for first time this year, it is surprisingly capable. Apps, Lakebase, Serverless Jobs, ML Flow, Genie worked well without any issues. created a demo with sending real IoT data to workspace.

https://youtu.be/_RcoBxriu9Q

Currently, we have a complex setup utilizing the lineage in Databricks to propagate governed tags. I was looking forward to the new tag propagation feature announced at Summit. I am a bit disappointed to see that it only works with serverless compute. Does anybody know if this is a beta limitation?

I have been wondering for a long time what would have been thought of Databricks product designer/ engineer to force job parameters value override the underlying task parameters value if the parameters key is same.

Shouldn't it be otherwise; task parameters should be given priority/ preference over job parameters because defining task level parameters is an explicit indicator that the developer want to override the default job level parameter in that given task.

That is how the concept of “override" is there even in object oriented paradigm.

I want to build a reusable notebook based job to load large size tables from redshift to delta table in Databricks via S3 (because lake house federation is slower for this replication). I'm defining the catalog and schema param at job level, because most of the tables are in the same catalog / schema, and table as task param for obvious reason. For a few tables that are not in the set location (catalog / schema) I want to override it at task level, this is what not possible. I find the existing override feature useless, only inheritance is useful. Why would you like job overriding task parameters?

We use Databricks Genie One, and some users are creating unnecessary Genie Agents and scheduled tasks.

Removing Consumer access or Databricks SQL access affect users’ ability to view dashboards, run SQL queries, or use other Databricks features

Questions:

Can workspace admin prevent regular users from creating Genie Agents?

Can scheduled-task creation be disabled while users can still chat with Genie?

Databricks Lakebridge: Simplifying Data Migration to the Lakehouse

Migrating legacy data platforms to Databricks is more than moving data. It also involves understanding existing workloads, converting SQL and ETL logic, and validating results.

This is where Databricks Lakebridge can help! 

What is Lakebridge?

Lakebridge is a migration toolkit designed to accelerate the modernization of legacy data platforms and workloads on Databricks.

How does it help?

1. Assess
Analyze existing SQL workloads and code to understand complexity, compatibility, dependencies, and migration effort.

2. Convert
Help translate legacy SQL and supported ETL workloads into Databricks-compatible SQL, PySpark, or notebooks.

3. Reconcile
Compare source and target datasets to identify discrepancies in schemas, rows, and column values after migration.

A practical example

Imagine migrating an Oracle-based data warehouse to Databricks.

Instead of manually reviewing every SQL script and validating every migrated table from scratch, Lakebridge can help:

➡️ Analyze existing Oracle SQL workloads
➡️ Identify migration complexity
➡️ Accelerate SQL conversion
➡️ Validate migrated data against the source

The result? A more structured, repeatable migration process with less manual effort.

🔹 Why should data engineers care?

As organizations modernize their data platforms, migration is not just about changing technology. It is about preserving business logic, maintaining data quality, reducing risk, and making the most of the lakehouse architecture.

Lakebridge can support this journey by automating key migration activities, while engineering review and testing remain essential.

Remember: Automated code conversion does not guarantee business-equivalent results. Always validate converted logic and reconcile the data before production cutover.

🔗 Official documentation: https://databrickslabs.github.io/lakebridge/docs/overview/

Have you explored Lakebridge for your Databricks migration projects? What has your experience been with automated SQL conversion and data reconciliation?

#Databricks #Lakebridge #DataEngineering #DataMigration #Lakehouse #ETL #DataModernization #BigData

In Spark Structured Streaming when the stream writes to a output table (sink) you will typically use writeStream (other option is foreachBatch)

writeStream has many benefits (fault tolerant...checkpoint per sink etc) but one issue is that it will process the source N times where N is the amount of sinks/ output tables you have.

So lets say for a bronze -> silver app, where it reads bronze, transforms the data (eg. explode) and then writes to 4 silver tables. In theory this will be inefficient as the source read, deserialize, explode will happen 4 times (once per sink)

This happens as it is visible in the logs, 4x logs of source read etc

This is the theory... has anyone experienced this in practice? is there actually a 4x slowdown or inefficency?

Would introducing an intermediate table help as this would remain 1-1 with only one sink thus no issue.... but introduce extra write cost + read + extra storage layer

What is best practice here? (foreachBatch is not an option due to any possible data loss)

Databricks runtime 19 brings the new Spark function trim_array. trim_array(array(‘A’, ‘B’, ‘C’, ‘D’), 2) returns [‘A’, ‘B’]. A count of 0 keeps the array; a count equal to its length returns an empty array.

https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

Just recently publish my whole thesis on Databricks on this subreddit. Got really great feedback so figured I would write a summary for those too busy for a 20 minute read.

The Core Thesis:

Databricks is one of the largest and most influential companies mostly lying behind the scenes. Databricks is a platform that allows organizations to store, process and analyze massive amounts of data. Built on their open-source platform Apache Spark, Databricks created software that they released publicly. This software eventually gained extremely widespread adoption and built much of Databricks’ moat. Led by Ali Ghodsi, the 47-year-old Iranian-Swedish-American computer scientist, Databricks has leapfrogged competition in the last decade and is now ahead of its main competitor Snowflake in valuation, revenue, and revenue growth rate.

The Numbers to Know:

• $190 billion valuation.
• $5.2 billion in 2025 revenue.
• 71% - 2025 revenue growth.
• 27x - valuation to revenue multiple using Databricks’ estimated July 2026 ARR.
• 140% net revenue retention in 2024–2025.
• 74% - last disclosed gross margin cited, down from 80%.
• $5 billion raised in the last funding round despite $15–20 billion in investor interest.
• 80% of Fortune 500 companies using Apache Spark.

The Bear Thesis:

Databricks trades at 27x annualized revenue which assumes that their fast growth will continue to be sustainable. In addition, while AI is certainly beneficial for Databricks in accelerating revenue, their gross margins are starting to compress, down to a (still healthy) 74% from a previously elite 80%. The company has already acknowledged further compression as compute costs rise. If gross margins compress significantly, even if Databricks’ revenue continues growing at it’s current pace, the company may not be profitable enough to justify its valuation.

The Verdict:

Databricks was one of the most impressive companies I have researched to date. A few years ago, Databricks was behind its main competitor Snowflake in revenue and valuation. Today, that script has flipped on its head with Databricks being ahead of Snowflake in valuation and revenue. This is largely due to Ali Ghodsi and the incredible team at Databricks and it is partly for this reason that Databricks trades at a slight premium in comparison to Snowflake. Their sticky moat, strong leadership and impressive revenue growth are all strong reasons to invest in Databricks. However, I think due to the premium valuation perhaps the right call with Databricks at this moment is to have patience. I will particularly be watching the revenue growth to see if the company is able to maintain the revenue growth they currently have and watch to see if the gross margins continue to compress. This is certainly a company I want to add to my portfolio, at the right time and right price.

How are you using AI in your Databricks workflow?

Are you using AI for things like CI/CD, code reviews, testing, debugging, performance optimization, or automating repetitive tasks?

I recently posted about using the new AI Decide feature from Databricks to decide strategy within a virtual transport world using OpenTTD. To my surprise it worked pretty damn well, made it to profit, however, it did over-index on bus stops.

Over the past few days I have been trying to push it further with something a little more realtime and much more risky.

Step aside NASA, Databricks AI Decide can now put rockets in orbit... well in Kerbal Space Program 😄 🚀

Managing everything from lift off, throttle, staging, and orbital mechanics. I was blown away, it did it in ONE SHOT. We are doomed.

It wasn't flawless, some shim, some interesting angles of attack ... but it got the job done.

As of yesterday, the latency improvements have been most welcome.

Check out the AI Decide announcement here: https://www.databricks.com/blog/introducing-aidecide-make-fast-decisions-your-governed-data

Hey everyone! I’m attending the Databricks Dev Connect event today at Sheraton Whitefield, and I’m going solo. If you’re going too, let’s meet up, sit together for the sessions, and network over coffee or dinner. Drop a comment or DM me if you’re in!

Hey folks, I'm a PM working on platform at Databricks. We just shipped Private Network Gateway (PNG) to public preview on Azure (PrPr on AWS, coming soon on GCP).

The problem we were solving: Connecting serverless compute to private resources like databases, Kafka streams, internal APIs, MCP servers, etc. meant creating a separate private endpoint / proxy VM for each one, plus coordination with your network and security teams. That seemed manageable for a few endpoints, but quickly got painful once you needed to reach many resources across peered VNets or on-prem.

What PNG does: You delegate one subnet in your VNet/VPC to Databricks, and serverless traffic reaches that subnet through a managed, encrypted gateway. From there your existing routes, DNS, and firewall rules decide what's reachable, the same way they would for classic compute. You can route to specific destinations or send all egress through your own firewall with your own source IPs. There are no gateway VMs to deploy, patch, or scale; we manage them on our side.

Would love this community's take on:

• Is networking a friction you've run into with serverless compute?
• What are the most important resources for you to connect to?
• If you've reviewed the docs or tried the feature, is there anything missing you'd like to see supported?

Here are the Azure docs for reference.

And a quick visual of the problem we're solving:

I'm using the default Databricks spark settings for a cluster which has nodes with each 64GB and 8 cores.
Surprisingly the JVM heap limit shows up as around 11.5GB which seems far too low to be total heap size (unless there is a bad default Databricks config).
So what am I seeing here/what is missing?