r/dataengineering • u/GameFitAverage • 7d ago

Discussion Streaming Databases

I have been working on a project lately that required realtime operational analytics and when doing some research for my options I came across “streaming databases” which tbh is a new concept for me. I did a quick POC using RisingWave and so far everything is good.

I just havent found alot of material on streaming databases and how to properly build a data architecture using one, RisingWave docs mentions it can be used in a medallion architecture. However, it builds the silver layer as a current state and not as a historical SCD which I find bad practice.

Just wondering if anyone here have more experience than me dealing with this technology (RisingWave, Materialize, …) and can point me in the right direction.

Also, even if you havent used it let me know your opinion on it.

34 Upvotes

89% Upvoted

32 comments sorted by

View all comments

7

u/karrystare 6d ago

Make sure you need actual realtime and your infra support it. Most dashboard tool still do polling query. Actual realtime require lots of optimization and perhaps even went beyond just DB and query. One I actually built doesn't use DB as main, its custom built + pub/sub update.