I get where you're coming from but I'd argue the kvm group is still better even when you automatically give all human users membership. You can then have less-privileged accounts for service roles, for example nginx doesn't need kvm acccess.

nginx might not, but some automated tool / cron job using libguestfs might. Kernel KVM has a very good record for security so it's not something to worry about, compared to (for example) lesser used drivers/filesystems or other dusty corners of the kernel.