Easier: Do not start with a "allow all" configuration in the first place.

Maybe all of those userspace-work-done-in-kernel-because-muh-performance features should be restricted to (the "real") CAP_NET_ADMIN, unless positively enumerated as free-for-all-containers. And then subtract from that free-for-all list every time you learn that some kernel module in its currently available version cannot be trusted to do its own memory shuffling.