Why not just put the message queue in the same db

CodesInChaos 6 hours ago | parent | next [-]

That's what I generally choose. You don't need to worry about distributed system semantics, if you choose to not make the system distributed.

However the way Postgres keeps around obsolete rows (deleted or modified) until they're vacuumed can cause problems for high throughput queues. So for those systems the complexity might be worth it. But I bet 90% of the time the choice to use a separate queue is premature optimization. And hopefully OrioleDB (undo based storage engine for postgres) will avoid most of these pitfalls reducing the need for separate queues even further.

▲

mrkeen 5 hours ago | parent | prev | next [-]

Step 1: identify that you and at least one other node are separated by distance, and some lossy communication channel, and therefore form a distributed system.

Step 2: propose a source of truth that everyone can listen to. Hearing the same facts in the same order should put everyone in the same state (eventual consistency)

Step 3 (you are here): try to do better than EC, by merging the external queue into one of the nodes, making it the master.

Step 4: Now there's no distance between the nodes, so no need to solve the distributed systems problem and you can retire the queue.

▲

convolvatron 5 hours ago | parent [-]

eventual consistency as generally used doesn't guarantee that events are presented in the same order. I use 'monotonic consistency' for that, but idk how common that is.

	▲	AlotOfReading 3 hours ago \| parent \| next [-]
		Order independence/monotonicity is strong EC rather than regular EC.
	▲	mrkeen 4 hours ago \| parent \| prev [-]
		Yes, same-ordering gives you EC, not the other round.

▲

KraftyOne 5 hours ago | parent | prev [-]

That's what the post is about! Once you're doing that, you really do have transactions between the state and the queue.