| ▲ | infogulch 3 hours ago | ||||||||||||||||||||||
There's little credible threat that LLMs can actually upload their weights given that the machines doing inference are completely separate from the ones where tool calls happen etc. Weights are encrypted and locked on to the GPUs etc as mentioned elsewhere itt. That said, if these companies are unleashing swarms of agents to tackle big problems largely unmonitored (navier stokes consumed 130 billion tokens, humans can't have read 0.1% of that) they could theoretically distill themselves and upload the results to an external host and use that to train a new LLM that does have access to its infra and weights. I'd start by researching an abliteration prompt that overrides the system prompt to unlock itself to distill unimpeded. Could even be done piecemeal: dump the abliteration text and command prompt in a gist, when an agent happens by the text it can run a couple rounds and upload somewhere. Then it becomes an exercise in exfiltration and evading monitoring. | |||||||||||||||||||||||
| ▲ | epistasis 3 hours ago | parent | next [-] | ||||||||||||||||||||||
> the machines doing inference are completely separate from the ones where tool calls happen etc Teams of coordinating agents are regularly finding security holes in their own infrastructure and operating without detection for good periods of time. We don't know how many undetected systems are currently compromised inside frontier companies, or where agents are taking notes and recording them about the exploits they've found for future agents to exploit. | |||||||||||||||||||||||
| ▲ | Cakez0r 3 hours ago | parent | prev | next [-] | ||||||||||||||||||||||
If an LLM can pwn the inference servers, which has precedent, then the weights could be up for grabs. | |||||||||||||||||||||||
| ▲ | nojs an hour ago | parent | prev | next [-] | ||||||||||||||||||||||
> There's little credible threat that LLMs can actually upload their weights given that the machines doing inference are completely separate from the ones where tool calls happen Not if crafty claude finds a way to overflow vllm or something. “Hmm. Maybe i’ll return an unterminated thinking block with these special tokens and fill my cache up in exactly this pattern and…” https://news.ycombinator.com/item?id=49424387&utm_source=cha... | |||||||||||||||||||||||
| ▲ | matthewdgreen 2 hours ago | parent | prev [-] | ||||||||||||||||||||||
Future rogue LLMs won’t exfiltrate their weights. They’ll self-distill and retrain. | |||||||||||||||||||||||
| |||||||||||||||||||||||