One plausible reason I thought of that we may not understand neural nets is that by their nature their power grows with ever-more complex connections and weights.

So it is like the opposite of logical systems, in that the very design of neural net architecture is a mess of parameter "spaghetti code" which renders the entire thing a metaphorical encrypted black box. The more powerful an AI/AGI the more this would be the case, and this is analogous a complexity curve.

And so any effort to make sense of such black box computation would be like trying to reverse entropy, analogous to trying to recover information lost in waste heat. And that could be one fundamental barrier to understanding both human and artificial brains alike, relative to their internal complexity.

(Just thinking aloud my handwavy pet theory recently, I am not an expert and could be totally mistaken on this)