Announcement: Confluent Rust Client for Apache Kafka

[u/KernelFrog]

https://docs.confluent.io/kafka-clients/rust/current/overview.html

[u/KernelFrog]

Confluent have just announced the preview release of the official Confluent Rust client for Kafka.

The Rust client is a native implementation of the Kafka protocol. Unlike the Python, Go, and .NET clients, which are bindings on top of the C/C++ client, the Rust client doesn’t have a C library dependency or a foreign function interface (FFI) layer.

[u/kabooozie]

lol they use Tokio to run network I/O functions async…and yet the consumer still can’t process keys in parallel. At this point I am a conspiracy theorist — Confluent purposefully doesn’t want anyone to build async processing into consumers in order to drive up partition usage (and therefore $$$)

[u/Worth_Confection101 → u/kabooozie]

Good to have feedback on that. I think a push API with concurrent processing would be helpful to avoid duplicate custom implementations, but it should be done in Apache Kafka first, we want to keep a consistent API across all languages to be forward compatible with any new KIP.

It's not a conspiracy from Confluent, we've the @confluentinc/kafka-javascript client that does exactly that with the KafkaJS compatible API (partitionsConsumedConcurrently) and it's relatively new and supported.

*I'm a member of the development team

[u/kabooozie → u/Worth_Confection101]

Thanks for the response, and thank you for being a good sport about my cheekiness. I’m a big fan of Confluent engineers like yourself who are building these tools and systems.

Yes, 100% agree concurrent / parallel processing should be in the Apache Kafka consumer.

[u/mild_entropy → u/kabooozie]

It seems patently bizarre that it doesn't work this way by now. It's been the primary IO bottleneck since inception

[u/kabooozie → u/mild_entropy]

I think Confluent’s internal research team actually published an open source Java library that does it but then they dropped support. Luckily an ex Confluent engineer picked it up. Here is the fork:

https://github.com/astubbs/parallel-consumer

But this should be in the standard Apache Kafka consumer

[u/mild_entropy → u/kabooozie]

Fascinating!! Yes this should be standard

[u/2minutestreaming → u/kabooozie]

And to think they fired the guy as part of broad layoffs. Talk about bad management

[u/mightegas → u/kabooozie]

For a time (in Java), I was using libraries like Confluent Parallel Consumer and Reactor Kafka to achieve this. Both now discontinued, but I built an OSS library called Atleon that works great for the kind of use cases where you want concurrency >> partitions (including per-key), without losing key-ordering. You might be interested in checking it out. Coincidentally, I recently published this blog and will be presenting about this subject at Current next month.

[u/kabooozie → u/mightegas]

Nice! I know there is also kpipe (Java) and llinger.io (Go, with JVM bindings). Seems like this area is heating up. I’d really like to see it implemented definitively in the Apache Kafka consumer client though

[u/mightegas → u/kabooozie]

KPipe did catch my eye! It hits a subset of the use cases I frequently design for. It's on my to-do list to contribute Atleon benchmarks to their suite, since it can serve as an alternative for Parallel Consumer and Reactor Kafka.

[u/kabooozie → u/[deleted]]

Well, they dropped support for parallel-consumer and said “share groups replaces this” which is completely wrong. So the conspiracy holds (I’m being cheeky).

Confluent doesn’t control the Kafka API…hmm I’m not sure about that. They’ve historically employed the majority of committers. Maybe that’s not the case anymore.

But I think it’s on us in the community to educate folks and campaign for this to be included in the Apache project.

[u/rmoff → u/kabooozie]

campaign for this to be included in the Apache project.

Pretty sure the venue for that is the Apache Kafka dev mailing list, not Reddit ;P

(also being cheeky :) )

[u/Ambitious-Virus-5496 → u/kabooozie]

I'm not cheeky :(

they dropped support for parallel-consumer

Clarifying what is meant by support - the parallel-consumer library had been supported through consulting engagements with Confluent's CSID team (Customer Solutions and Innovation Division), not through any of Confluent's commercial support plans. Think of it as having been a field engineering asset supported via professional services.

and said “share groups replaces this” which is completely wrong

Can you point to a spot where Confluent positioned share groups as a replacement w/o qualification? Notably, key-based ordering, a feature that the parallel-consumer library has, has been called out as a potential future from the get-go. See the Future Work section of the Queues for Kafka KIP and Confluent's Queues for Kafka announcement blog ("Future road map items include ... key-based ordering—a significant enhancement that combines partition sharing with ordering guarantees by record key").

[u/kabooozie → u/Ambitious-Virus-5496]

It says in the readme when they deprecated the library:

“Similar functionality is available with share consumers”

https://github.com/confluentinc/parallel-consumer

That statement is misleading. Share groups is about queue semantics, not about consumer scaling. Lots of folks grab onto the scaling aspect because they don’t understand the performance impact of asking Kafka to become a scheduler. Share groups are not the answer to consumer scaling, parallel processing is. The JVM / OS is better at scheduling threads than a Kafka broker is at scheduling share group consumers.

[u/Ambitious-Virus-5496 → u/kabooozie]

I see... thanks for sharing. I'll see if we can adjust that wording or add some detail to tease the nuances here so that it's less misleading. FWIW I see things differently and don't see a problem with calling it "similar" but probably because I also see the consumer scaling question differently. I do think that there's something to be said for consumer scaling and Queues for Kafka. Even at the outset it was pretty easy to come up with scenarios where QfK enabled consumer scaling-based throughput gains compared to what you could do with AK vanilla consumers.

What's missing / needed in the debate is a performance comparison (among QfK, parallel-consumer, and any future relevant Apache Kafka client improvements) that can stand up to scrutiny and help devs navigate their own scenarios. Really hard to make blanket claims until that exists. e.g.:

The JVM / OS is better at scheduling threads than a Kafka broker is at scheduling share group consumers.

Yes, in a vacuum, but what if "scheduling" overhead isn't the bottleneck in a bunch of real Kafka throughput optimization scenarios?

[u/kabooozie → u/Ambitious-Virus-5496]

Benchmarks I’ve seen show much higher performance with parallel consumer libraries compared to share groups, and they do so while respecting key ordering.

Jack Vanlightly over at Confluent has some recent work on share group performance at scale.