You can apply memory quotas to the individual processes with cgroups. You can also adjust how likely a process is to be killed.