| ▲ | burcsahinoglu 2 hours ago | |
I had a terrible time with a cluster of RabbitMQ servers, the cluster was partitioning often, impacting the application. It seemed like it was very sensitive to transient network problems. At the end we moved from a cluster to a single instance running on a over provisioned machine, that worked pretty well. | ||
| ▲ | jorl17 38 minutes ago | parent [-] | |
At a previous job (many years ago) we used to run RabbitMQ. I hadn't configured it originally, but I was certainly left with it in pieces when it entered split-brain mode in the face of some bizarre network failure. So we had our application talking to two nodes of a cluster, both working normally without talking to each other (nevermind the fact that the connectivity between the machines was fine). Some workers picked up jobs, others didn't. IIRC the app read requests from a queue and wrote replies to another one, so we'd have half of the jobs being picked up, and then half of the replies not being picked up (ergo, 75% of the requests failed). It was bizarre and nothing seemed to indicate there were issues with RabbitMQ itself. I don't even remember how we figured it out at the time. It's been nearly 10 years, I'm sure that it's better, that I was inexperienced, that the original setup wasn't great. But that left me so sore on it that I have actively avoid it ever since. Bonus points for having dealt with this while Django Celery apparently had two independent configuration mechanisms, and both were being used in this app. Fun times. | ||