I'm not a mathematician but to me it doesn't seem so far-fetched to think that there might exist some mathematical proof that ensures the indistinguishability

We don’t have the ability to do that level of analysis of natural language text mathematically.

If we did, we probably wouldn’t need LLMs in the first place, i.e. we could just generate text using explicitly programmed algorithms.