Training the weights of the neural network produces a humungous function with a vast number of parameters.

Such a function is not inherently mysterious due to the size alone. For instance, if we fit a billion numeric points to a polynomial curve having a billion coefficients, we would not be mystified as to how the polynomial interpolates between the points.

Be that as it may, the trained neural network function does have mysterious properties, that is true.

But that doesn't mean we don't know how it works. We invented it and produced it by training.

To say that we completely don't understand it is like saying we don't understand thermodynamic because the laws of thermodynamic don't allow us to predict the path of a particle of gas in, and so we must remain mystified as to how the gas can take on the shape of the container.

Say we train a neural network to recognize digit characters. Of course we know why it produces the answer 3 when given any one of our training images of 3: we iterated on bumping the weights until it did that. When we give it a an image of 3 not in our training set and it produces some answer (either correctly 3 or something disappointing) we are less sure. We don't exactly know the exact properties of the multi-dimensional function which encode the "threeness" of the image.

Sure; so what? It's a heck of a lot more than we know about how a person recognizes a 3, where we had no design input, and don't even know the complete details of the architecture. We don't have a complete model of just one neuron, whereas we do have a complete model of a floating-point number.

Gas in a container is a kind of brain which figures out how to mimic the shape of the container using a function of a vast number of parameters governing the motion of particles. Should we be mystified and declare that we don't understand the thermodynamic laws we came up with because they don't track the path taken by a particle of gas, and don't explain how every particle "knows" where it is supposed to be so that the gas takes on the shape of the cylinder, and has equal pressure everywhere?

> we would not be mystified as to how the polynomial interpolates between the points.

We would not be surprised - we wouldnt know how the model resolve the problem. We wouldn't know if it is approximating, calculating the correct value, or memorising result. We would only know how it was built. We would be mystified in how it solved the problem.

> But that doesn't mean we don't know how it works. We invented it and produced it by training.

It is not because it was invented that we know how it works. The fallacy in your reasoning is thinking that Emergent Behavior or Properties can be trivialy explained, by knowing it's building block.