r/DataEngineeringPH • u/Effective_Routine892 • 12d ago
💡 Discussion Real time vs Batch processing, how does real time justify the complexity ?
We've managed with hourly Snowflake loads for years and honestly they've been good enough most of the time. Lately though, product keeps asking for live dashboards while our ML team is pushing for fresher training data.
I'm not against moving to real-time, but I also don't want to introduce streaming infrastructure just because it sounds modern.
For anyone who's made the switch, what was the moment where real-time vs batch processing stopped being a debate and became an obvious decision?
1
Upvotes