I would love to know if there's a way to secure this though. I'm not prepared to have people constantly trying to login to my immich instance so it's only accessible via VPN

You can use something like Immich Public Proxy to only expose the /share path of your server and keep the main /api path that has everything else behind VPN