We already don't understand LLMs and they are having oberservable effects

If some LLM somewhere managed to produce a coherent and _correct_ explanation of how it worked inside, but no human ever saw it, would it matter?

We know how to apply LLms to problems, which is a subtly different thing to understanding how they do what they do. It's similar to fire: I can cook using fire, but I don't really understand how fire _works_. Heat+oxygen+fuel, sure, but what goes on chemically? I dunno. Doesn't stop me using it. (Pretty sure _humanity_ knows how fire works, though).