> When I program, my application may sometimes allocate a lot of memory due to some silly bug. In such case the whole system practically stops working [...]

You can limit resource usage per process thus your buggy application could be killed long before the system comes to a crawl. See your shell' s entry on its limit/ulimit built-in or use

man prlimit(1) - get and set process resource limits