Nothing about providing the API for this requires that your phone automatically accept such requests.

They can change it so that all first requests require confirmation with password.

And you can secure your HTTP server and SSH server with credentials too, but if your wanting a secure internal service it’s also probably a good idea to put a firewall in place and only allow access from authorized endpoints. Security isn’t a binary thing. It comes in layers and “no publicly accessible API” is (or at least can be) more secure than “public API”.