OS are designed to fully exploit available resources, Linux tries its best before triggering an OOM kill.

I recommend using the earlyoom if you want more aggresive oom kill:

https://github.com/rfjakob/earlyoom

The README contains a lot of interesting information.

A strange behavior I sometimes run into with earlyoom is that I try to start up some buggy software of mine and it seemingly never starts.

It took a long evening to figure out that it gets earlyoom'd immediately because it tries to allocate too much. Previously the very familiar hitching and freezing was a very easy sign of what kind of issue I was dealing with