Can I suppose that Cursor, by seeing everyones prompts and all of the responses from the models, are in a position to sort of MITM distill them?

Musk was forced to admit under oath during his OpenAI lawsuit that xAI distilled GPT models to train Grok. No reason to believe he won't use Cursor data for the same thing, if it isn't happening already.

Yes, and given it's now owned by Elon Musk, who is not trustworthy, it's possible he would try to do it.

Although I wouldn't take an Openai announcement literally, as Sam Altman is also not trustworthy.

Unless you run your models and harness completely locally, you can't verify your data isn't being used.

All of these companies, and I truly mean all, stole data to train their models.