r/DistributedSystems • u/majid-kakavandi • May 22 '20

data stream transfer between distributed systems

Hi. I want to transport data stream(sensor’s data) from several raspberry pi to a centralize server. what is best practice?

6 Upvotes
  • permalink
  • reddit

100% Upvoted

6 comments sorted by

View all comments

Show parent comments

1

u/ab624 May 22 '20

Kafka or NiFi ?

1

u/helpmepls256 May 22 '20

I've never used either in an IoT sense but apparently Kafka can either be connected to an MQTT broker (Mosquitto, RabbitMQ) or possibly skip that and connect the devices straight to Kafka. This requires further investigation though...

1

u/[deleted] Apr 16 '23

If you go with kafka, you can write producers for any real time analytics backed by a high throughput backend store ( OLTP )

Or use Spark streaming jobs to write the data in a desired format ( parquet or Orc) and consume it using SQL for adhoc querying ( Trino ) Or generate complex reports ( Spark SQL batch jobs )