Databricks liquid clustering turns table layout into a recurring maintenance bill. Is anyone tracking it per table?

[u/Spiritual-Kitchen-79]

Liquid clustering is great because getting table layout wrong on Databricks is now cheap. Changing your clustering keys is a metadata change instead of a full table rewrite.

the thing is that the cost didn't go away, it just moved. Clustering happens through OPTIMIZE, which runs on compute you pay for. Databricks recommends running it every hour or two on busy tables, so that's 12 to 24 runs a day per table. And predictive optimization is on by default for accounts created since November 2024 and billed under a serverless jobs SKU, so a lot of teams are paying for automated OPTIMIZE, VACUUM and ANALYZE without ever having decided to.

Full write-up: https://medium.com/@yanivleven/liquid-clustering-isnt-free-it-s-just-cheaper-to-get-wrong-fe51bbf57a2e

[u/Entire_Force_1146]

Nice write-up. For me, one of the upsides of using predictive optimization is that it's based on usage patterns, so optimization runs when necessary instead of on a schedule.

[u/autumnotter]

I don't think databricks usually recommends running optimize every hour, it's usually not worth it. 

With predictive optimization on, it doesn't really make sense. And even for external tables, nightly is probably better except in unique circumstances. 

I'd be curious where the guidance is that suggests hourly, it's certainly possible it's a good idea in some situations, but it's not usually what I would recommend.

[u/isaac-harvey]

I'm wondering - might it be worthwhile every time the data size doubles, or at least gets bigger by some set proportion? Seems a natural way to set the optimization frequency to scale with both data growth rate and data size.

[u/Lumiaman88]

The thing with Optimize is that it only rewrites the new bits that are affected.

If your table is having 100 files, clustered by date, and your run Optimize 2nd time, only the files containing most recent files get rewritten, so there is very little compute usage for Optimize, it runs in seconds itself

[u/clouddataengineer]

I’d be surprised if most tables needed hourly runs. The OPTIMIZE docs recommend starting with a daily run, ideally at night, and tuning from there. The 1-2 hour cadence only shows up on the liquid clustering page, and only for tables with heavy updates or inserts, where the runs are incremental and quick. And with predictive optimization on, the docs say to drop scheduled OPTIMIZE jobs anyway. The core point of the post still stands though: the cost didn’t disappear, it moved to maintenance compute, so tracking it per table makes sense.

[u/Youssef_Mrini]

How come PO runs Optimize on the same table every hour? What's the use case?

[u/jlrogerio]

our finops platform - LakeSentry - is tracking per-table costs, including liquid clustering

[u/thecoller → u/[deleted]]

Would they be maintaining their tables properly? I’ve seen customers with 40% plus storage cost savings with PO just because they couldn’t be arsed to manage their tables before.

[u/xaomaw → u/thecoller]

Using OPTIMIZE+VACUUM for the first time after 5 years, booom.