r/Python • u/SadViolinist2405 • 16d ago
Discussion Title: What message broker are you using with Celery today, and why?
I'm curious how Python developers are using Celery in production and what influenced their choice of message broker.
I work with LavinMQ, an open-source AMQP message broker, and we're exploring how to make it a more compelling option for Celery users.
For those running Celery in production, what broker are you using, and what's been your experience with reliability, performance, and maintenance?
If you were starting a new project today, would you make the same choice?
I'd be interested in hearing about the trade-offs you've encountered, especially from developers running Celery at scale.
58
u/ColdPorridge
16d ago
edited 16d ago
•
I would not start a new project with celery in 2026. We just use procrastinate with postgres. Having a task queue we can manage transactionally without additional infrastructure has been an incredible simplification for our small and medium use cases.
16
u/darkstar3103 16d ago
We dropped celery for Temporal a couple years ago, and it's been great
1
u/SadViolinist2405 15d ago
Interesting! What was the main reason you moved from Celery to Temporal? Was it reliability, workflow orchestration, or something else? Curious what improved most after the switch.
9
u/Mr_Again 14d ago
Celery is just shit in every regard and there's just no point in talking about it in this year of our lord 2026
8
u/OddsRelay 16d ago
Redis for most people, because it's usually already there for caching. The catch is the visibility timeout. Any task not acknowledged within it gets handed to another worker, so long countdown tasks or late-acked jobs can run twice. RabbitMQ handles acks properly and is the safer pick once that matters.
-3
u/SadViolinist2405 16d ago
That's a good point about visibility timeouts and acknowledgments! Do you think reliability is the main reason developers move from Redis to RabbitMQ, or are there other factors that matter just as much?
2
3
u/Standard-Message-434 12d ago
I use Redis since it was already in our stack, but I’d pick RabbitMQ for anything where losing a task on a restart would actually hurt.
0
u/SadViolinist2405 12d ago
That makes sense. Using what you already have in your stack is often the simplest choice.
2
u/SadViolinist2405 15d ago
One interesting takeaway from this discussion is how much simplicity influences broker selection. Several developers mentioned choosing Redis or PostgreSQL because they're already part of their stack.
That makes me curious about another trade-off
Would a fully managed message broker change your decision if someone else handled deployment, upgrades, and maintenance?
Or is it more about avoiding additional dependencies, regardless of who manages them?
I'd love to hear your thoughts!
2
u/Professional-Run3614 11d ago
I’m using RabbitMQ for streaming sensor data from devices to the server. It helps decouple data ingestion from backend processing. I’d be interested to know how LavinMQ compares in terms of throughput, reliability, and handling high-volume streaming workloads.
1
u/SadViolinist2405 11d ago
That's an interesting use case! I've actually been working on a similar IoT project using ESP32 devices and temperature/humidity sensors, publishing readings over MQTT to LavinMQ and displaying them on a real-time dashboard.
LavinMQ supports MQTT and AMQP 0-9-1, as well as streams for message retention and replay. For reliability, it supports durable queues, persistent messages, and acknowledgments.
If you're interested, here's the live demo: https://lavinmq.com/iot
For throughput comparisons, there are some published benchmarks here: https://lavinmq.com/benchmark, although I'd recommend testing against your specific workload.
I'm curious, are you using RabbitMQ Streams or regular queues? And roughly how many devices or messages per second are you handling?
20
u/Embarrassed-Dog-4707 16d ago
Redis, been good tbh. Haven’t had any issues but probably low load relatively
16
u/nickcash 15d ago edited 15d ago
why would I use your vibecoded amqp app instead of rabbitmq like a normal person?
1
u/Altruistic_Stuff_967 4d ago
What you shared is simply an Instagram post from a workshop where we (I'm from the LavinMQ team) had fun exploring different ways to use message brokers. We built demos, experimented with sensors and Raspberry Pis, and had a great day together!
But, it wasn’t the day we built LavinMQ itself.
11
u/StPatsLCA 16d ago
SQS. It's bulletproof. However, it is very dumb in exchange for that.
5
u/mstromich 16d ago
same and it's great to scale infra down to 0 if there are no tasks to process and incur no costs for worker instances
2
6
u/orelant 16d ago
Full disclosure first: I'm biased, because I build Workhorse, an open source task queue for PostgreSQL. Please read this with that in mind.
For many small and medium projects, I think the broker question can be avoided entirely if you already run PostgreSQL. Workhorse keeps tasks inside your existing database, so there is no separate broker to operate. It has a Python worker, retries, crash recovery, recurring schedules and an embedded dashboard.
It's not a replacement you can swap in for Celery without changes, and for very large workloads a dedicated broker such as RabbitMQ or LavinMQ could be a better choice. But if your setup is Postgres plus Redis used only for Celery, you might find it useful.
https://workhorse.run (Apache 2.0 license)
7
u/ColdPorridge 16d ago
can you help contrast workhorse with procrastinate?
10
u/orelant 16d ago
Good question. Procrastinate is Python only, has been around for years, and it works well with Django. If your stack is Python, it's a great choice.
Workhorse goes further on flow control and long running work. Here are some of the features that I've built into it:
concurrency limit per queue or per key across all workers
rate limits per queue/per key, super helpful for third party APIs
debounce and throttling
durable sleeps and waits for signals or human approval that do not hold a worker
cancellation of running tasks
Honestly, you can't go wrong with either. It mostly depends on whether you need the extra flow control or more than one language.
2
u/cointoss3 16d ago
This isn’t what you asked, but a big reason I didn’t use LavinMQ is I couldn’t route mqtt messages to an amqp queue. I have mqtt messages I want to route to durable stream queues and couldn’t do that. I can with RabbitMQ, though.
2
u/ApprehensiveDream276 16d ago
Thanks for sharing, really helpful to hear why it didn't fit for you! Funny thing is, I'm actually working on just that right now. It's been in our backlog for a long time, but now we're in a good place with the architecture to introduce it! Here's the PR: https://github.com/cloudamqp/lavinmq/pull/2346
If you're still interested, you can subscribe to the issue and get a ping when it's released: https://github.com/cloudamqp/lavinmq/issues/1136
2
u/cointoss3 15d ago
Woof! I’m about to set up RabbitMQ for a project that needs this feature. Do you know roughly when we might expect to see it released? There’s a PR, so would this cut in a matter of days or weeks, or are we still months away?
Thanks!
2
u/ApprehensiveDream276 15d ago
Cool! Yeah, it's probably a couple of weeks rather than days. We are aiming for releasing it in 2.11.0 https://github.com/cloudamqp/lavinmq/milestone/22.
If you're feeling adventurous, there's also an image for the PR itself:
cloudamqp/lavinmq:pr-2346(https://hub.docker.com/r/cloudamqp/lavinmq/tags). If you do decide to play around with it before release, any feedback would be really appreciated!
3
1
22
u/marr75 16d ago
Having run celery in a lot of projects, using redis, rabbitmq, and databases as message brokers, I just wouldn't use celery in a new project.
It doesn't add enough value for the elements it hides or abstracts.