This is an essential use case for XDP - this is how FB's firewall works, and above that their LB uses the same technology.

The beauty of XDP is that it's all eBPF. Completely customizable by injecting policy where it's needed and native to the kernel.