There's a field known as information geometry. I don't know much about it myself as I'm more into physics, but here's a recent example of applying geometrical analysis to neural networks. Looks interesting as they find a phenomenon analogous to phase transitions during training

Information Geometry of Evolution of Neural Network Parameters While Training

https://arxiv.org/abs/2406.05295