▲ | followben 17 hours ago | |||||||
How does this compare to other pg-backed python job runners like Procrastinate [0] or Chancy [1]? | ||||||||
▲ | gabrielruttner 13 hours ago | parent | next [-] | |||||||
Gabe here, one of the hatchet founders. I'm not very familiar with these runner so someone please correct me if I missed something. These look like great projects to get something running quickly, but likely will experience many of the challenges Alexander mentioned under load. They look quite similar to our initial implementation using FOR UPDATE and maintaining direct connections from workers to PostgreSQL instead of a central orchestrator (a separate issue that deserves its own post). One of the reasons for this decision to performantly support more complex scheduling requirements and durable execution patterns -- things like dynamic concurrency [0] or rate limits [1] which can be quite tricky to implement on a worker-pull model where there will likely be contention on these orchestration tables. They also appear to be pure queues to run individual tasks in python only. We've been working hard on our py, ts, and go sdks I'm excited to see how these projects approach these problems over time! [0] https://docs.hatchet.run/home/concurrency [1] https://docs.hatchet.run/home/rate-limits | ||||||||
| ||||||||
▲ | wcrossbow 16 hours ago | parent | prev | next [-] | |||||||
Another good one is pgqueuer https://github.com/janbjorge/pgqueuer | ||||||||
▲ | INTPenis 17 hours ago | parent | prev [-] | |||||||
Celery also has postgres backend, but I maybe it's not as well integrated. | ||||||||
|