I know it’s what the security folk think about, exfiltrating to a model endpoint is the least of my concerns.

I work on commercial OSS. My fear is that it’s exfiltrated to public issues or code. It helpfully commits secrets or other BS like that. And that’s even ignoring prompt injection attacks from the public.

In the end if the data goes somewhere public, it'll be consumed and in today's threat model another GenAI tool is going to exploit faster than any human will.