How do you actually learn concurrency without fooling yourself?

Every time I think I “get” concurrency, a real bug proves otherwise.

What finally helped wasn’t more theory, but forcing myself to answer basic questions:

What can run at the same time here?

What must be ordered?

What happens if this suspends at the worst moment?

A rough framework I use now:

First understand the shape of execution (what overlaps)

Then define ownership (who’s allowed to touch what)

Only then worry about syntax or tools

Still feels fragile.

How do you know when your mental model is actually correct? Do you rely on tests, diagrams, or just scars over time?

I've written, tested and debugged low-level java concurrency code involving atomics, the memory safety model and other nasty things. All the way down to considerations if data races are a problem or just redundant work and similar things. Also implementing coroutines in some complang-stuff in uniersity.

This level is rocket science. If you can't tell why it is right, you fail. Such a failure, which was just a singular missing synchronized block, is the _worst_ 3-6 month debugging horror I've ever faced. Singular data corruptions once a week on a system pushing millions and trillions of player interactions in that time frame.

We first designed with many smart people just being adverse and trying to break it. Then one guy implemented, and 5-6 really talented java devs reviewed entirely destructively, and then all of us started to work with hardware to write testing setups to break the thing. If there was doubt, it was wrong.

We then put that queue, which sequentialized for a singular partition (aka user account) but parallelized across as many partitions as possible live and it just worked. It just worked.

We did similar work on a caching trie later on with the same group of people. But during these two projects I very much realized: This kind of work just isn't feasible with the majority of developers. Out of hundreds of devs, I know 4-5 who can think this way.

Thus, most code should be structured by lower-level frameworks in a way such that it is not concurrent on data. Once you're concurrent on singular pieces of data, the complexity explodes so much. Just don't be concurrent, unless it's trivial concurrency.

Heisembugs aren't just technical debt but project killer time bombs so one must better have a perfect thread design in head that works first attempt, else is hell on earth. I can be safe in a bubble world with whole process scope individual threads or from a thread pool (so strong guarantees of joining every created thread) and having share-nothing threads communicating only by prosumer sync-queues that bring a clear information-flow picture. One can have a message pump in one thread, as GUI apps do. That is just a particular case of the prosumer channel idea before. Avoid busy waits, wait on complex event conditions by blocking calls to select() on handler-sets or WaitForMultipleObjects(). Exceptions are per thread, but is good to have a polite mechanism to make desired ones to be potentially process-fatal, and fail earliest. This won't cover all needs but is a field-tested start.

Share xor mutate, that's really all there is