For production workloads, you can use systemd socket activation to avoid most of the network issues. The caddy demo I've linked below explains more about the issues it would solve.

[0] https://github.com/containers/podman/blob/main/docs/tutorial...

[1] https://github.com/eriksjolund/podman-caddy-socket-activatio...