Why is celery awful?

> The Many Problems with Celery:

— https://steve.dignam.xyz/2023/05/20/many-problems-with-celer...

> The problems with (Python’s) Celery:

— https://docs.hatchet.run/blog/problems-with-celery

> Dramatiq motivation:

— https://dramatiq.io/motivation.html

Here are some alternatives:

Dramatiq: https://github.com/Bogdanp/dramatiq

RQ: https://github.com/rq/rq

Huey: https://github.com/coleifer/huey

Hatchet: https://github.com/hatchet-dev/hatchet

It's okay till it's not. Everyone I know who had Celery in production was looking for a substitution (custom or third-party) on a regular basis. Too many moving pieces and nuances (config × logic × backend), too many unresolved problems deep in its core (we've seen some ghosts you can't debug), too much of a codebase to understand or hack. At some point we were able to stabilize it (a bunch of magic tricks and patches) and froze every related piece; it worked well under pressure (thanks, RabbitMQ).

Because it’s a seducer. It does what you need to do and you two are happy together. So you shower more tasks on Celery and it becomes cold and non-responsive at random times.

And debugging is a pain in the ass. Most places I’ve been that have it, I’ve tried to sell them on adding Flower to give better insight and everyone thinks that’s a very good idea but there isn’t time because we need to debug these inscrutable Celery issues.

https://flower.readthedocs.io/en/latest/

Celery is great and awful at the same time. In particular, because it is many Python folks' first introduction to distributed task processing and all the things that can go wrong with it. Not to mention, debugging can be a nightmare. Some examples:

- your function arguments aren't serializable - your side effects (e.g. database writes) aren't idempotent - discovering what backpressure is and that you need it - losing queued tasks during deployment / non-compatible code changes

There's also some stuff particular to celery's runtime model that makes it incredibly prone to memory leaks and other fun stuff.

Honestly, it's a great education.