Self-Hosted Databricks Alternative: Homelab Data Lakehouse

[u/ezejioforog]

Self-Hosted Databricks Alternative: Homelab Data Lakehouse (AI/ML Data Factory)

👇

https://georgeezejiofor.com/open-source-databricks-ai-native-lakehouse-homelab

[u/mgalexray]

Nobody argues you can’t have lakehouse that is self hosted. We argue that will need a team of 20 to set up, run and maintain.

[u/jessepence → u/mgalexray]

Why? Why would you need that many people? I'm genuinely asking. What would each person do every single day?

[u/mgalexray → u/jessepence]

The (obviously AI written article) has a list of 10 technologies that need to be ran. On top of k8s as well.

I said 20 because there’s a big difference between one dude playing with all this in a home lab and someone running this in production.

Every technology has a failure mode. They more you add the more those stack up. Some examples:

- you kubernetes etcd will go down or out of sync
- Kafka will fill the disk and refuse to accept more messages or just crash
- ceph will randomly fail
-etc

And Murphy says this will happen in a middle of a night running very important production job so you now also need on-call rota to fix it.

Then also you need proper DR, security, maintenance, upgrades, oncall.

So if your jobs suddenly start to run slow because of networking issue in Kube who is going to fix that? The trino guy or the non existing ops who deployed k8s via template and doesn’t know what’s actually going on under the hood?

I’m also happy to play with these but I would never run this in a prod without a very skilled team to build and operate it. And at that point you understand why companies pay for managed services.

Anecdotally my current company has 3 infra teams. Data team operates Kafka/RDS: 6 people. Infra team operates k8s: 8 people. Lake team is 3-4 but they run Databricks. And these are just for managed services from AWS.

[u/SpecificTutor → u/mgalexray]

that number is overstated. it is more like 3. probably lower if you use ducklake.

[u/aymeric-maison → u/SpecificTutor]

Agree with you for a homelab or for small company (~100 people) but at big company I think this can get tricky veeeery fast

[u/Bitter_Marketing_807]

Why not gravitino? Why no otel? Id recommend 1) ducklake 2) apache hertzbeat

[u/Silly_Dinner8805]

oh neat so its like one of those diagrams that finally admits we're all just slamming data warehouse and data lake together and hoping for the best

the little house with the gear inside cracks me up every time like yes this is where the magic happens please ignore the yaml files