It scares me too.

I actually think this is precisely the reason LLMs can't be the basis for a technological revolution. Because it's only one way.

Like, if you have a compiler, and it has a bug. You can discover if that bug is influencing your code execution and patch it. You can go both up and down the stack.

With LLMs, there is no way to patch it's translation function. You have to rely on it to forward process.

I don't think there is any way to avoid us understanding our tech stacks.