Maybe not in kernel, but running the earlyoom daemon will let you do exactly that in userspace.