For any given set of model weights and inputs? Yes, we definitely do understand them.

Do we understand the emergent properties of almost-intelligence they appear to present, and what that means about them and us, etc. etc.?

No.

Right. The machine works as designed and it's all assembly instructions on gates. The values in the registers change but not the instructions.

And it happens to do something weirdly useful to our own minds based on the values in the registers.