> You can 100% predict where the weights “will take you” given a set of inputs.

Do you mean reproduce?

Sorry it's just if you are saying what your statement implying then either the model is very simple, or you've figured out something incredible

Sure. Take the inputs and run it through the model.

You now have the output which will be perfectly reproduced with the same inputs.