The best part is if you consider it a vulnerability, it is one you can't fix.

It reminds me of SQL injection techniques where you have to exfiltrate the data using weird data types. Like encoding all emails as dates or numbers using (semi) complex queries.

If the L(L)M has the data, it can provide it back to you, maybe not verbatim, but certainly can in some format.