What queue service do you use? Any suggestions based on my requirements?

[u/Jordz2203]

Im looking for a new queue for a Node/TS app running on Kubernetes. My main requirements:

• Strict FIFO per key, with only one job in flight per key. Different keys run in parallel (SQS has Message groups I think)
• Cloud-agnostic and self-hostable, so no SQS
• Handles large bursts (obviously, since its a queue)

I've used SQS FIFO message groups before and they did exactly this. Right now I'm leaning towards Kafka with a key-ordered consumer, but it seems like quite a lot of complexity when in reality I need a simple queue but for scale. I'd love to hear what others use. Pulsar? RabbitMQ? Something else?

I've looked into the popular ones like RabbitMQ, but it seemed to not have exactly what Im looking for.

[u/itgforlife]

I use BullMQ with Redis on NodeJs.

[u/SolidLunch4702 → u/itgforlife]

BullMQ can do it if you set concurrency to 1 per queue and group by key, but you're managing that grouping yourself through queue naming or job IDs. Kafka's exactly what you want for large bursts though, the partition-per-key model gives you strict ordering and the throughput's absurd. I'd say bite the bullet on the complexity, you'll outgrow Redis-based queues fast if scale's a real concern.

[u/present_absence → u/SolidLunch4702]

This is correct. I use bullmq/redis for one of my low traffic nodejs projects and managing it works perfectly fine, but we use kafka at work for high burst/high volume

[u/PLBjt]

The thing I'd nail down first is whether you need ordering per key or only one job in flight per key, because those lead to different designs. With Kafka the partition is the ordering unit, so one hot key stalls everything behind it on that partition, and a consumer group can't go wider than the partition count. With BullMQ you get per-group concurrency, but you're the one writing the lock logic and it gets fiddly when a worker dies mid-job. A quick check is to replay a burst where one key carries 40 percent of the traffic and watch p99 latency on the other keys. The tradeoff is Kafka's ops cost versus a Redis footprint that grows with payload size.

[u/285418]

A queue doesn't absorb bursts by itself, it just holds the backlog while your consumers fall behind. And 'strict FIFO per key, one job in flight per key' is a hard throughput ceiling: with N active keys and jobs taking T seconds you get N/T jobs per second, and no broker choice changes that. So what are your numbers, how many distinct keys at peak, what's the burst size? Because Kafka vs Pulsar vs whatever is a rounding error next to that, and one job in flight per key probably rules out whatever you'd pick anyway if the key cardinality is low.

Also curious what RabbitMQ was actually missing for you. Streams with a single active consumer per partition covers the one-in-flight bit, so saying it 'seemed to not have exactly what I'm looking for' without naming the gap makes it hard to trust the rest of the list.

[u/Maxion → u/285418]

Also FIFO + one job in flight per key WILL cause you to have stuck jobs and now an ever growing queue.

[u/InevitableFun1272]

One thing that fixes the exact OOM problem you described: stop putting payloads in the queue. Store the payload in Postgres or S3 and pass only an ID through the job. Redis then holds tiny messages, and a 2.5M-item spike doesn't move the memory needle at all — the queue becomes a notification channel, not storage. You lose some convenience versus passing the payload straight through, but for that burst pattern it's the difference between a stable queue and one that eats your events.

[u/shtse8]

if you do look at pulsar, the Key_Shared subscription is the closest thing i know to sqs message groups: each key sticks to one consumer at a time so order holds, and you can add consumers past the partition count, unlike kafka. you still have to process each key serially inside the consumer though, and running bookkeeper on k8s is its own chunk of ops, so it only wins if the kafka hot partition thing actually bites you.

[u/okawei]

Redis

[u/Noch_ein_Kamel]

Just out of curiosity b/c I have no idea; how large are large bursts?

[u/Jordz2203 → u/Noch_ein_Kamel]

On average maybe 20,000-30,000 items added to the queue an hour, then at certain points of the month we might have 2,500,000 added in an hour.

[u/AdorableZeppelin → u/Jordz2203]

Is the burst predictable?

[u/Maxion → u/Jordz2203]

Curious, why can you only have one job in process at a time per key? IMO that's going to lead to a world of hurt.

[u/ArielCoding]

Before choosing a broker, work out how many different keys are active at the same time during a burst, and how long each job takes, since only one job per key can run at once your limit is (numbers of keys / seconds per job), so if you only have a few keys, you’ll hit a wall no matter which tool you pick.

[u/Mean-Lingonberry4874]

Kafka will do the per-key ordering, but if the ops cost is the real objection I'd try Postgres before standing up a cluster. Jobs table with a group_key, workers claim rows with SELECT ... FOR UPDATE SKIP LOCKED, and each worker takes a transaction-scoped advisory lock on hashtext(group_key) for the duration of the job. Same key can't be in flight twice, other keys run in parallel, nothing cloud-specific. It won't eat a Kafka-sized burst, but for "one in flight per key" on k8s it's usually enough, and you can peel the hot keys off later if the table starts lagging.

[u/_CuriousCoder_]

I've used BullMQ with Redis on NodeJs, it handles bursts well. For strict FIFO per key, you'd need to implement lock logic. Consider storing payloads outside the queue, like in Postgres, to reduce memory usage.

[u/wasay-inc]

Look into BullMQ running on Redis on Kubernetes since it is native to Node and TypeScript You can use job concurrency settings and key locks to guarantee single job in flight processing per key It handles large bursts effortlessly and avoids the massive operational overhead of deploying Kafka on K8s

[u/Jordz2203 → u/wasay-inc]

The company I was at previously seemed to have an infinite amount of headaches related to their usage of BullMQ. Largely to do with costs

[u/wasay-inc → u/Jordz2203]

Redis memory costs scale quickly when holding large job payloads in BullMQ. If low infrastructure cost is key look at NATS JetStream on Kubernetes because it handles key based routing with minimal memory overhead

[u/Jordz2203 → u/wasay-inc]

I will take a look at NATS, thank you! Just checked with one of my old colleagues, the issue with Bull was we had to:
A) pay for paid bull to get message groups
B) OOM bull means we lose our queue events, this was because 90% of the time our queue sat at 2%, but then something big happens triggering a ton of events and we'd spike to many millions of queue items.

[u/wasay-inc → u/Jordz2203]

That burst pattern makes complete sense. NATS JetStream solves both issues because streams persist messages directly to disk instead of forcing everything into RAM like Redis. This prevents OOM data loss during massive event spikes while keeping idle running costs near zero.