A first look at Django's new background tasks

JakaJancar 4 hours ago | parent | next [-]

Assuming you're fine with keeping the queue in postgres, I've used Procrastinate and it's great:

https://procrastinate.readthedocs.io/en/stable/index.html

Core is not Django-specific, but it has an optional integration. Sync and async, retries/cancellation/etc., very extensible, and IMO super clean architecture and well tested.

IIRC think the codebase is like one-tenth that of Celery.

	▲	TkTech 3 hours ago \| parent [-]
		If you like Procastinate, you might like my Chancy, which is also built on postgres but with a goal of the most common bells and whistles being included. Rate limiting, global uniqueness, timeouts, memory limits, mix asyncio/processes/threads/sub-interpreters in the same worker, workflows, cron jobs, dashboard, metrics, django integrations, repriotization, triggers, pruning, Windows support, queue tagging (ex: run this queue on all machines running windows with a GPU, run this one on workers with py3.14 and this one on workers with py3.11) etc etc... https://tkte.ch/chancy/ & https://github.com/tktech/chancy The pending v0.26 includes stabilizing of the HTTP API, dashboard improvements, workflow performance improvements for workflows with thousands of steps and django-tasks integration.

▲

frankhsu 3 hours ago | parent | prev | next [-]

Really cool to see a batteries‑included option in Django for background jobs.

For folks who’ve used Celery/Procrastinate/Chancy: how does retry/ACK behavior feel in real projects? Any rough edges?

What about observability — dashboards, tracing, metrics — good enough out of the box, or did you bolt on extra stuff?

Also, any gotchas with type hints or decorator-style tasks when refactoring? I’ve seen those bite before.

And lastly, does swapping backends for tests actually feel seamless, or is that more of a “works in the demo” thing?

	▲	TkTech 2 hours ago \| parent [-]
		(I'm biased, I'm the author of Chancy) One of the major complaints with Celery is observability. Databased-backed options like Procastinate and Chancy will never reach the potential peak throughput of Celery+RabbitMQ, but they're still sufficient to run millions upon millions of tasks per day even on a $14/month VPS. The tradeoff to this is excellent insight into what's going on - all state lives in the database, you can just query it. Both Procastinate and Chancy come with Django integrations, so you can even query it with the ORM. For Chancy in particular, retries are a (very trivial) plugin (that's enabled by default) - https://github.com/TkTech/chancy/blob/main/chancy/plugins/re.... You can swap it out and add whatever complex retry strategies you'd like. Chancy also comes with a "good enough" metrics plugin and a dashboard. Not suitable for an incredibly busy instance with tens of thousands of distinct types of jobs, but good enough for most projects. You can see the new UI and some example screenshots in the upcoming 0.26 release - https://github.com/TkTech/chancy/pull/58 (and that dashboard is for a production app running ~600k jobs a day on what's practically a toaster). The dashboard can be run standalone locally and pointing to any database as-needed, run inside a worker process, or embedded inside any existing asgi app.

▲

vforgione 6 hours ago | parent | prev | next [-]

I’ve been using the django-tasks library in production for about a year. The database backend and simple interface have been great. It definitely isn’t intended to replace all of celery, but for a simple task queue that doesn’t require additional infrastructure it works quite well.

	▲	benatkin an hour ago \| parent [-]
		This one? https://github.com/RealOrangeOne/django-tasks That and the rq backend sound promising to me.

▲

matsemann 7 hours ago | parent | prev | next [-]

How is the typing support? We just had downtime because a change to a celery task didn't trigger mypy to complain for all call sites until runtime. Too many python decorators aren't written with pretty weak typing support.

▲

roam 6 hours ago | parent | next [-]

With regards to args and kwargs? None. Your callable is basically replaced with a Task instance that’s not callable. You need to invoke its enqueue(*args, **kwargs) method and yeah… that’s of course not typed.

▲

halfcat 6 hours ago | parent | prev [-]

Static analysis will never be fully robust in Python. As a simple example, you can define a function that only exists at runtime, so even in principle it wouldn’t be possible to type check that statically, or even know what the call path of the functions is, without actually running the code in trace/profiler mode.

You probably want something like pydantic’s @validate_call decorator.

	▲	tomjakubowski 4 hours ago \| parent [-]
		> you can define a function that only exists at runtime, so even in principle it wouldn’t be possible to type check that statically Can you say more, maybe with with an example, about a function which can't be typed? Are you talking about generating bytecode at runtime, defining functions with lambda expressions, or something else?

▲

ethagnawl 4 hours ago | parent | prev | next [-]

This is an exciting development. I imagine I'll continue using Celery in most cases but being able to transparently swap back-ends for testing, CI, etc. is very compelling.

I haven't looked into this in any detail but I wonder if the API or defaults will shave off some of the rough edges in Celery, like early ACKs and retries.

▲

the__alchemist 6 hours ago | parent | prev | next [-]

This is great! The prev recommendation was usually a lib called celery that I wasn't able to get working. I don't remember the details, but it had high friction points or compatibility barriers I wasn't able to overcome. This integration fits Django's batteries included approach.

I've been handling this, so far, with separate standalone scripts that hook into Django's models and ORM. You have to use certain incantations in an explicit order at the top of the module to make this happen.

	▲	halfcat 6 hours ago \| parent [-]
		> scripts that hook into Django's models and ORM Django has management commands for that [1]. When you use Django over time, you experience this pleasant surprise over and over when you need something, “oh, Django already has that” [1] https://docs.djangoproject.com/en/5.2/howto/custom-managemen...

▲

fud101 4 hours ago | parent | prev [-]

Django this is about 10 years too late. It's frustrating because we use all manner of hacks to work around this being part of the builtin story.