As usual, prepare for Linux suggestions to the papercuts, but...

I use a program called earlyoom. It will monitor RAM and if you cross a level of utilization (default 95%?) it will kill the worst offenders before the system becomes unresponsive. You can layer on sophistication like protecting certain programs or preferring killing others. I find it invaluable when I am doing data science work and do something stupid which explodes in memory. Annoying that something was killed, but usually better than hosing the entire system -if it crossed 95% it was almost certainly going to hit 100%.

For my purposes it works perfectly - only the Python process will be killed, my IDE or notebook will survive.

Thank you so much for proposing earlyoom, that is exactly what I needed!