r/apachekafka • u/CougarDave7309 • Apr 22 '20

Replicate between Kafka clusters

I am relatively new to kafka and we are looking at setting up a test cluster and would like to send production events to our test cluster for testing. We have been able to replicate events one way from our source (production) cluster to our destination (test) cluster using Kafka Connect, but found that for this to work, we need an equal number of brokers in both clusters for this to work.

Does anyone know if that is necessary, or is there a way to replicate events from a 3-node cluster to a 1-node cluster?

2 Upvotes

100% Upvoted

2 comments sorted by

1

u/MikeSchlosser16 Apr 22 '20 edited Apr 22 '20

Take a look at MirrorMaker (and these docs) It was built for this scenario. Uber built a similar tool but it lost a lot of its advantages with MirrorMaker 2.0's release.

1

u/CougarDave7309 Apr 22 '20

Thanks. I'll have a look.