I've written, tested and debugged low-level java concurrency code involving atomics, the memory safety model and other nasty things. All the way down to considerations if data races are a problem or just redundant work and similar things. Also implementing coroutines in some complang-stuff in uniersity.

This level is rocket science. If you can't tell why it is right, you fail. Such a failure, which was just a singular missing synchronized block, is the _worst_ 3-6 month debugging horror I've ever faced. Singular data corruptions once a week on a system pushing millions and trillions of player interactions in that time frame.

We first designed with many smart people just being adverse and trying to break it. Then one guy implemented, and 5-6 really talented java devs reviewed entirely destructively, and then all of us started to work with hardware to write testing setups to break the thing. If there was doubt, it was wrong.

We then put that queue, which sequentialized for a singular partition (aka user account) but parallelized across as many partitions as possible live and it just worked. It just worked.

We did similar work on a caching trie later on with the same group of people. But during these two projects I very much realized: This kind of work just isn't feasible with the majority of developers. Out of hundreds of devs, I know 4-5 who can think this way.

Thus, most code should be structured by lower-level frameworks in a way such that it is not concurrent on data. Once you're concurrent on singular pieces of data, the complexity explodes so much. Just don't be concurrent, unless it's trivial concurrency.

That’s why the actor model is so good. You have concurrent programs but each Actor has full ownership of its data and can access it as if it were single threaded. In my opinion, it’s the only way to get it right and should only be replaced with low level atomics if performance proves to be much better and that impacts the business strongly, which I have never seen in practice.

I'm interested in knowing more details about this if you happen to have a post written up somewhere!