I think we need the equivalent of general relativity for latent spaces.

This is the field of information geometry: https://en.wikipedia.org/wiki/Information_geometry