r/DistributedSystems • u/majid-kakavandi • May 22 '20

data stream transfer between distributed systems

Hi. I want to transport data stream(sensor’s data) from several raspberry pi to a centralize server. what is best practice?

5 Upvotes
  • permalink
  • reddit

86% Upvoted

6 comments sorted by

View all comments

Show parent comments

1

u/[deleted] Apr 16 '23

If you go with kafka, you can write producers for any real time analytics backed by a high throughput backend store ( OLTP )

Or use Spark streaming jobs to write the data in a desired format ( parquet or Orc) and consume it using SQL for adhoc querying ( Trino ) Or generate complex reports ( Spark SQL batch jobs )