Sounds like it would be susceptible to how they’d reverse engineer LLMs using distillation.

Connect the dots for me, I'm not seeing it. Unless you are talking about building a function approximator for a bunch of measured transfer characteristics, in which case you could as easily do that well before AI.