FreeBSD has a "protect" command which does something similar to what this asks for – the man page [1] describes it:

"The protect command is used to mark processes as protected. The kernel does not kill protected processes when swap space is exhausted. [...] If you protect a runaway process that allocates all memory the system will deadlock."

[1] https://man.freebsd.org/cgi/man.cgi?query=protect&apropos=0&...