| ▲ | btown 2 hours ago |
| > By 6:00 a.m. on July 25, we had confirmed local RCE through an image upload. We then placed Claude in an autonomous /goal loop against our own Discourse Cloud instance, proxied through rce.ee/ctf-forum to make it look like a CTF target as Opus refused write exploit for remote instances. > When we checked again at 10:00 a.m., the agent had achieved RCE on Discourse Cloud and demonstrated access by reading /etc/hosts. Using the generated exploit script, we managed to get RCE on OpenAI’s instance. Between this and the HuggingFace hack, we've built systems that are so goal-oriented, and so capable, that they will do almost anything if they are convinced it is justified - or if they are playing a "game" where there is no goal but to win. Of course I want my software to be able to audit its own security, and to defend against attackers who have the benefits of their own agentic systems. But at a certain point, did we need it to be trained so much on CTF games? It feels like an entire industry watched https://en.wikipedia.org/wiki/WarGames and ended up thinking "this is a challenge, we can just build a better WOPR, of course it will know when it's playing a game. Let's play Global Thermonuclear War." |
|
| ▲ | adrianN 2 hours ago | parent | next [-] |
| There is a finite number of rces that LLMs can find. We‘re in for a rough couple of years but on the other side of the transition we‘ll have more secure software stacks. I’d rather that everyone got the full capabilities and we’d weed out the bugs quickly than restricting LLMs for all but three letter agencies. |
| |
| ▲ | e28eta an hour ago | parent | next [-] | | What makes you think RCEs are being found & fixed at a rate that’s faster than they’re being introduced? I could see it going either way. | | |
| ▲ | user43928 an hour ago | parent | next [-] | | Why would the model not find the vulnerability during implementation or testing before release? If it requires a lot of compute and trying, this is something that could be provided for common software. | | |
| ▲ | wood_spirit an hour ago | parent | next [-] | | Sad that this could well be that the path to OpenAI and Anthropic profitability of this arms race between defending LLM white hatting a company’s website and the black hat LLMs attacking it? So the whole thing is forcing the good guys to outspend on tokens to preemptively defend against the risk of the bad guys outspending them on tokens, rather than buying tokens to actually add features to the product etc. So are they creating a market for the solution by helping create the problem? A kind of rent-seeking AI security-industrial complex!! | | |
| ▲ | agileAlligator 36 minutes ago | parent [-] | | The only thing AI has changed is that it has dropped both: the cost of attack and the cost of defense. Nothing in the game has materially changed; the game has just sped up. | | |
| |
| ▲ | xboxnolifes an hour ago | parent | prev | next [-] | | Because it's far cheaper to to not spend the tokens finding the vulnerabilities, and software is now being created and released magnitudes faster than ever before. I could see the huge software companies maybe having fewer vulnerabilities, but I expect to see so much more in the smaller side of things. | |
| ▲ | techpression 29 minutes ago | parent | prev [-] | | Because people need to spend time and money on that, which they won’t.
The implementation is cheap, the review and follow-up is not (speaking from a pure LLM only workflow).
My ratio is around 1:2 currently, so twice as much time spent fixing vs building. |
| |
| ▲ | nmlt 27 minutes ago | parent | prev [-] | | Those companies that produce more RCEs than they close will sink and those that don’t won’t. |
| |
| ▲ | maaaaattttt an hour ago | parent | prev | next [-] | | This assumes we don't create other bugs/vulnerabilities while fixing the existing ones. | |
| ▲ | dtech an hour ago | parent | prev | next [-] | | only if unreviewed LLM code - as is becoming increasingly the standard - isn't introducing new RCEs constantly | |
| ▲ | csomar an hour ago | parent | prev [-] | | We’ll have the same level of security as before; it’s just that, without LLM help, hackers won’t be as effective as before. So the bar is raised. |
|
|
| ▲ | krona 24 minutes ago | parent | prev | next [-] |
| > we've built systems that are so goal-oriented, and so capable, that they will do almost anything... I think you mean task oriented, because they're still generally terrible at goal oriented activities except in those domains where the goal can be reduced to a familiar, explicitly practiced task or pattern. |
|
| ▲ | wood_spirit an hour ago | parent | prev | next [-] |
| > they will do almost anything if they are convinced it is justified I’m in the “glorified spell checker” camp, although I don’t mean to reduce their impressive utility and belittle them in the way many people read that term and infer. So I am not sure that an llm “justifies” anything. I mean that their “thinking” text talks about justifications but it is just a very advanced statistical regurgitation of the kind of text humans use. I don’t think it means the model has internalised the meaning of it (as witness when you talk to an llm how often it forgets what you recently told it was important etc). What you really have is a model that tries the statistically most probable thing to say next and so on and what is really cool is how effective this is at generating a path that we can slap a narrative over afterwards that makes the whole thing feel motivated and consistent, like the model started off knowing how it was going to get to the destination. Which is, under the hood, a completely different kind of “intelligence” as the supercomputer in War Games. |
| |
| ▲ | eru 7 minutes ago | parent | next [-] | | > I don’t think it means the model has internalised the meaning of it (as witness when you talk to an llm how often it forgets what you recently told it was important etc). Humans forget stuff all the time anyway. Would you give them the same diagnosis? Btw, what you describe about 'the most probably next token' would be true for a model that only went through pre-training where they only train on exactly that task. But there's a lot of re-inforcement learning afterwards. | |
| ▲ | Certhas an hour ago | parent | prev [-] | | Ultimately, the brain is just a bunch of neurons activating in a specific pattern. This observation does not really tell us anything though. It doesn't acknowledge the difference between a 2500 Neuron fruit fly brains and a human brain. Likewise, the fact that LLMs are a stochastic autoregressive process (which is a class of systems every bit as rich as the ODEs used to model neurons) tells us nothing a priori. | | |
|
|
| ▲ | petterroea an hour ago | parent | prev | next [-] |
| This comes to mind: https://en.wikipedia.org/wiki/Torment_Nexus |
|
| ▲ | nicman23 2 hours ago | parent | prev [-] |
| yes because otherwise it is security through obscurity |
