I guess you have to use firewall as well. So basically you block any access from internet except VPN service. And you can have rule which IP allowed to access your VPN service.