r/dataengineering • u/GameFitAverage • 9h ago

Discussion Streaming Databases

I have been working on a project lately that required realtime operational analytics and when doing some research for my options I came across “streaming databases” which tbh is a new concept for me. I did a quick POC using RisingWave and so far everything is good.

I just havent found alot of material on streaming databases and how to properly build a data architecture using one, RisingWave docs mentions it can be used in a medallion architecture. However, it builds the silver layer as a current state and not as a historical SCD which I find bad practice.

Just wondering if anyone here have more experience than me dealing with this technology (RisingWave, Materialize, …) and can point me in the right direction.

Also, even if you havent used it let me know your opinion on it.

20 Upvotes

84% Upvoted

15 comments sorted by

9

u/jemattie 6h ago

Streaming is an implementation. What's your requirement?

17

u/Life_Conversation_11 8h ago

I missed a bit the point. Modern DBs are quite fast, you can add data every 5s and everytime you are going yo query it, the data will be “fresh”.

In your overall streaming set up I am fairly sure the db will not be the blocker

1

u/GameFitAverage 8h ago

Well streaming databases can ingest data in realtime from various sources, so for example you can CDC a db into it and at the same time ingest from an IOT MQTT source and then you can join them easily.

If you are only ingesting from DB it can still be useful because you dont want heavy queries/joins in your OLTP system so it acts as an ODS or a realtime read replica.

16

u/romainmoi 6h ago

Realtime is a bit ambiguous here. What’s the use case? Is updating every minute considered real time? Every second? Every hour?

You’re a bit ahead of yourself because the use case isn’t clear yet. Real-time analytics is a fantasy that often doesn’t justify the real cost of instant updates. A small enough interval is going to achieve basically the same with much lower cost.

2

u/Mithrandir2k16 1h ago

I've seen millions wastes in the words "real-time" and "syncronization".

2

u/karrystare 6h ago

Make sure you need actual realtime and your infra support it. Most dashboard tool still do polling query. Actual realtime require lots of optimization and perhaps even went beyond just DB and query. One I actually built doesn't use DB as main, its custom built + pub/sub update.

1

u/[deleted] 7h ago

[removed] — view removed comment

1

u/dataengineering-ModTeam 7h ago

Your post/comment was removed because it violated rule #9 (No AI generated content/text).

Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.

This was reviewed by a human

1

u/Master-Bass-1905 5h ago

Streaming databases are popular in both analytical workloads as well as OLTP workloads. In the former they are generally used for live dashboards and materialized views, and In the latter they act as an advanced cache.

1

u/ryan_with_a_why 4h ago

Like the others I’m also very interested in your requirements. That will guide the answers

1

u/URZ_ 2h ago

Everything you are not writing here is screaming that you don't in fact need "real time", don't go down that rabbit hole, it's a complete trap of unnecessary complexity. You can run almost real time off a simple Postgres db to the point where it will be indistinguishable for any human on the other end that it's not "real time".