A First Look at Django's New Background Tasks

Static analysis will never be fully robust in Python. As a simple example, you can define a function that only exists at runtime, so even in principle it wouldn’t be possible to type check that statically, or even know what the call path of the functions is, without actually running the code in trace/profiler mode.

You probably want something like pydantic’s @validate_call decorator.

With regards to args and kwargs? None. Your callable is basically replaced with a Task instance that’s not callable. You need to invoke its enqueue(args, *kwargs) method and yeah… that’s of course not typed.

> scripts that hook into Django's models and ORM

Django has management commands for that [1].

When you use Django over time, you experience this pleasant surprise over and over when you need something, “oh, Django already has that”

[1] https://docs.djangoproject.com/en/5.2/howto/custom-managemen...

Celery generally works best with the redis backend.

The main thing when it comes to models is that you pass the ids not the model instances themselves when passing to celery jobs.

This is because the celery worker could be running somewhere else entirely.

This one? https://github.com/RealOrangeOne/django-tasks

That and the rq backend sound promising to me.

Mostly people used Celery which has been around almost as long as Django.

The best time to plant a tree is twenty years ago. The second best time is now.

I also warmly recommend procrastinate !

We moved all our celery tasks to procrastinate at work for all our django backends since almost two years now and it has been great.

Having tasks deferred in the same transaction as the business logic stuff is something that helped us a lot to improve consistency and debugability. Moreover, it's so nice to able to inspect what's going on by just querying our database or just looking at the django admin.

For those wondering, procrastinate has no built-in alternative to django-celery-beat, but you can easily build your owned in a day: no need for an extra dependency for this :)

If you like Procastinate, you might like my Chancy, which is also built on postgres but with a goal of the most common bells and whistles being included.

Rate limiting, global uniqueness, timeouts, memory limits, mix asyncio/processes/threads/sub-interpreters in the same worker, workflows, cron jobs, dashboard, metrics, django integrations, repriotization, triggers, pruning, Windows support, queue tagging (ex: run this queue on all machines running windows with a GPU, run this one on workers with py3.14 and this one on workers with py3.11) etc etc...

https://tkte.ch/chancy/ & https://github.com/tktech/chancy

The pending v0.26 includes stabilizing of the HTTP API, dashboard improvements, workflow performance improvements for workflows with thousands of steps and django-tasks integration.

(I'm biased, I'm the author of Chancy)

One of the major complaints with Celery is observability. Databased-backed options like Procastinate and Chancy will never reach the potential peak throughput of Celery+RabbitMQ, but they're still sufficient to run millions upon millions of tasks per day even on a $14/month VPS. The tradeoff to this is excellent insight into what's going on - all state lives in the database, you can just query it. Both Procastinate and Chancy come with Django integrations, so you can even query it with the ORM.

For Chancy in particular, retries are a (very trivial) plugin (that's enabled by default) - https://github.com/TkTech/chancy/blob/main/chancy/plugins/re.... You can swap it out and add whatever complex retry strategies you'd like.

Chancy also comes with a "good enough" metrics plugin and a dashboard. Not suitable for an incredibly busy instance with tens of thousands of distinct types of jobs, but good enough for most projects. You can see the new UI and some example screenshots in the upcoming 0.26 release - https://github.com/TkTech/chancy/pull/58 (and that dashboard is for a production app running ~600k jobs a day on what's practically a toaster). The dashboard can be run standalone locally and pointing to any database as-needed, run inside a worker process, or embedded inside any existing asgi app.

A customer of mine has two projects. One running on their own hardware, Django + Celery. The other one running on AWS EC2, Django alone.

In the first one we use Celery to run some jobs that may last from a few seconds to some minutes. In the other one we create a new VM and make it run the job and we make it self destroy on job termination. The communication is over a shared database and SQS queues.

We have periodic problems with celery: workers losing connection with RabbitMQ, Celery itself getting stuck, gevent issues maybe caused by C libraries but we can't be sure (we use prefork for some workers but not for everything)

We had no problems with EC2 VMs. By the way, we use VirtualBox to simulate EC2 locally: a Python class encapsulates the API to start the VMs and does it with boto3 in production and with VBoxManage in development.

What I don't understand is: it's always Linux, amd64, RabbitMQ but my other customer using Rails and Sidekiq has no problems and they run many more jobs. There is something in the concurrency stack inside Celery that is too fragile.

What were the problems you had with Celery?

I remember working at a startup in the early 2010's that used celery for all of its background job infrastructure. There were several million tasks run daily across dozens of servers. Celery would regularly hang. Queues would pile up. We had some crazy scripts that would restart celery, detect and kill hanging processes, etc. Fun times.

We switched from Celery to Temporal. Temporal is such a great piece of distributed system.

Can share the sentiment, had to work with celery years ago, and the maintenance/footguns exceeded the expectations. The codebase and docs are also a bit messy, it's a huge project used and contributed by many so it's understandable I guess. Anyway, Argo if you are in K8S, something else if you aren't. And if you are a startup and need speed, just go with something like procrastinate.

Migrated Celery to Argo Workflows. No wisdom as it was straightforward. You lose a lot startup speed though, so it's not a drop-in replacement and is only a good choice for long-running workflows. Celery was easier than Argo Workflows. Celery is really easy to get started with. I like Airflow the best, but it's closer to Argo Workflows in terms of more long-lived workflows. I hope to try Hatchet soon. I've read Temporal is even harder to manage.

It sounds like the need you’re describing is not event-driven, but more like orchestration where the orchestrator is aware of dependencies between tasks and runs them in the right order (it builds a DAG, a directed acyclic graph). Tools like Airflow do this.

They're not mutually exclusive. Nothing about "event driven" means async. I have an event driven modular monolith and all events are handled synchronously. It's up to the receiver to queue a task if it needs to, so context boundaries are not crossed.

To implement progress reporting, it means you are able to know the time a task would take to run upfront, no? Is it even possible to do it accurately ?

Though, I imagine you could have strategies to give an approximation of it, for example like keeping track of the past execution time of a given type of task in order to infer the progress of a currently running task of the same type.

I had previously asked about this. My understanding is that there is a desire to add functionality to allow this, but in this first release the task system is intentionally very lightweight.