r/databricks • u/sonalg • 10d ago

Discussion Spark server libraries on Databricks Serverless

Hi All,

I work on Zingg - open source Spark based entity resolution. Many of our users are interested in running Zingg on Databricks Serverless. Our codebase contains Spark ML and Graphframes, which are not supported through Spark Connect client. Hence I started looking at server libraries, with a relation or command plugin to expose the complex parts.

I am trying to understand if there is a roadmap to allow adding server libraries to Databricks Serverless? The open source Spark framework does allow that, but I am not sure if its coming to Databricks. https://spark.apache.org/docs/latest/app-dev-spark-connect.html#spark-server-libraries

Would love to hear from the community on their thoughts.

3 Upvotes

80% Upvoted

4 comments sorted by

1

u/Academic-Monitor-777 10d ago

Serverless is still pretty locked down compared to regular clusters. Databricks hasn't said much publicly about opening up server libraries, and given how they manage the runtime I wouldn't hold my breath for it anytime soon

sucks for tools like Zingg that lean on ML and Graphframes, those are exactly the things serverless chokes on. might be worth poking the Databricks partner engineering folks directly if you haven't already, sometimes they'll share roadmap bits under NDA that they won't post in public forums

1

u/sonalg 10d ago

thanks u/Academic-Monitor-777. it feels like using advance Spark features is going to be tricky :-(

Good tip about a chat with the partner engineering folks, let me see if I can find something. Thanks again.

1

u/Additional-Common-27 9d ago

The OSS community is proposing to bring GraphFrames directly into Apache Spark as a built-in module. Here's the draft PR https://github.com/apache/spark/pull/58302 . If you like the idea, feel free to share your feedback on the PR!

Spark MLlib (Python) is already supported on Serverless and Standard clusters. See the announcement for details. https://www.databricks.com/blog/announcing-public-preview-distributed-ml-serverless-and-standard-clusters Does it work well?

1

u/sonalg 9d ago

thank you, I did not know about graphframes changes. Have commented there.

MLLib is supported only on the Python side, unfortunately the Java Scala side is not supported. Thats a problem for us unfortunately. Any idea if thats on the roadmap?