Remix.run Logo
antirez 6 hours ago

Every time some production environment can be simplified, it is good news in my opinion. The ideal situation with Rails would be if there is a simple way to switch back to Redis, so that you can start simple, and as soon as you hit some fundamental issue with using SolidQueue (mostly scalability, I guess, in environments where the queue is truly stressed -- and you don't want to have a Postgres scalability problem because of your queue), you have a simple upgrade path. But I bet a lot of Rails apps don't have high volumes, and having to maintain two systems can be just more complexity.

byroot an hour ago | parent | next [-]

> The ideal situation with Rails would be if there is a simple way to switch back to Redis

That's largely the case.

Rails provide an abstracted API for jobs (Active Job). Of course some application do depend on queue implementation specific features, but for the general case, you just need to update your config to switch over (and of course handle draining the old queue).

watercolorblind 3 hours ago | parent | prev | next [-]

The primary pain point I see here is if devs lean into transactions such that their job is only created together with the everything else that happened.

Losing that guarantee can make the eventual migration harder, even if that migration is to a different postgres instance than the primary db.

yawboakye 5 hours ago | parent | prev [-]

the problem i see here is that we end up treating the background job/task processor as part of the production system (e.g. the server that responds to requests, in the case of a web application) instead of a separate standalone thing. rails doesn’t make this distinction clear enough. it’s okay to back your tasks processor with a pg database (e.g. river[0]) but, as you indirectly pointed out, it shouldn’t be the same as the production database. this is why redis was preferred anyways: it was a lightweight database for the task processor to store state, etc. there’s still great arguments in favor of this setup. from what i’ve seen so far, solidqueue doesn’t make this separation.

[0]: https://riverqueue.com/

runako an hour ago | parent | next [-]

SolidQueue uses its own db configuration.

> it shouldn’t be the same as the production database

This is highly dependent on the application (scale, usage, phase of lifecycle, etc.)

erispoe 2 hours ago | parent | prev | next [-]

> it shouldn’t be the same as the production database

Why is that?

zarzavat an hour ago | parent [-]

If you need to restore the production database do you also want to restore the task database?

If your task is to send an email, do you want to send it again? Probably not.

stavros an hour ago | parent [-]

It's not like I'll get a choice between the task database going down and not going down. If my task database goes down, I'm either losing jobs or duplicating jobs, and I have to pick which one I want. Whether the downtime is at the same time as the production database or not is irrelevant.

In fact, I'd rather it did happen at the same time as production, so I don't have to reconcile a bunch of data on top of the tasks.

andrewstuart 4 hours ago | parent | prev [-]

It’s not necessary to separate queue db from application db.

yawboakye 3 hours ago | parent [-]

got it. is it necessary, then, to couple queue db with app db? if answer is no then we can’t make a necessity argument here, unfortunately.

nick__m 3 hours ago | parent | next [-]

Frequently you have to couple the transactional state of the queue db and the app db, colocating them is the simplest way to achieve that without resorting to distributed transactions or patterns that involve orchestrated compensation actions.

jrochkind1 an hour ago | parent | prev [-]

solid_queue by default prefers you use a different db than app db, and will generate that out of the box (also by default with sqlite3, which, separate discussion) but makes it possible, and fairly smooth, to configure to use the same db.

Personally, I prefer the same db unless I'm at a traffic scale where splitting them is necessary for load.

One advantage of same db is you can use db transaction control over enqueing jobs and app logic too, when they are dependent. But that's not the main advantage to me, I don't actually need that. I just prefer the simplicity, and as someone else said above, prefer not having to reconcile app db state with queue state if they are separate and only ONE goes down. Fewer moving parts are better in the apps I work on which are relatively small-scale, often "enterprise", etc.