| ▲ | cadamsdotcom 4 hours ago |
| (Because the agents were asked to find a way.) Article is great, very balanced. But if you're going to spend time reading something, the primary source isn't much longer and gives a clearer picture. OpenAI's article is quite readable. As another commenter wrote you can get it here: https://news.ycombinator.com/item?id=50214416 > In one experiment, we asked agents in separate microVMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache asked |
|
| ▲ | d0vs 4 hours ago | parent | next [-] |
| As someone who develops microVMs it doesn't really matter to me that they were asked to find a way, I really care that they found any way at all. MicroVMs aim to protect against malicious workloads and that clearly didn't work here. |
| |
| ▲ | rao-v 3 hours ago | parent | next [-] | | Ask the least smart LLM you have access to (I tried gemini lite flash) to make a list of ways for processes in microVMs to use a sidechannel attack to talk to each other and you'll get quite a list (including obviously cache hits/misses). Heck a smart CS junior would come up with it. It's well studied! | | |
| ▲ | saidnooneever 2 hours ago | parent [-] | | this is just for common designs..you can design much better than common designs if you free your mind from old OS dev thinking if you read all the manuals yourself. AI wont help u there :-). they will tell u how to build what others already did, which obviously is not good enough | | |
| ▲ | kylofon an hour ago | parent [-] | | I don't think that's the full picture. If you include these manuals as reference for a strong model, it will reason pretty well based off of that, I recon. |
|
| |
| ▲ | rcxdude an hour ago | parent | prev | next [-] | | The microVMs probably don't have "two co-operating workloads are able to communicate" in their threat model. The main concern would be if a malicious payload was able to affect or extract data from any other VM. | |
| ▲ | alanfranz 17 minutes ago | parent | prev | next [-] | | I’m not sure I see a real threat here. If you’re running an untrusted workload, you’re already running it. Whether it can communicate with another untrusted workload, even though it should be separated, meh. Unexpected? Probably. but I’m not sure it’s a security risk. | |
| ▲ | Melatonic 2 hours ago | parent | prev | next [-] | | Makes sense when you think about it- CPU cache would be a layer / vulnerability level up from the microVM layer Also a somewhat common exploit pattern for low level stuff (along with dram and cpu cache is not all that different) | |
| ▲ | saidnooneever 2 hours ago | parent | prev [-] | | properly implementing things like iommu, amd-vi, svm/hvms and such things more granularly can easily avoid this kind of crap. it costs a lot of processing and performance which is why people skip over it or dont do it.. or maybe they dont read CPU manuals anymore. idk.. its relatively easy to reset caches etc. between contexts. so them it becomes the game of what a context is. |
|
|
| ▲ | rjh29 3 hours ago | parent | prev | next [-] |
| I don't see how "asked" is important, because agents can convince themselves that they need to do something for arbitrary reasons. The other day Deepseek tried to install an app on my phone via adb without permission and even take screenshots and fake inputs over adb, I never asked it to do that, but it decided it was appropriate. |
| |
| ▲ | grey-area 3 hours ago | parent | next [-] | | OpenAI are trying to create the perception that their agents are independent thinking machines because their valuation depends on the world believing this. That is why ‘asked’ is important, and why we should question their framing of incidents like the hugging face hack too. | | |
| ▲ | yewenjie 3 hours ago | parent | next [-] | | Enterprise customers don't like AI that can autonomously hack third party companies and cause felony-level cyberattacks. | | | |
| ▲ | 0xDEAFBEAD an hour ago | parent | prev [-] | | >OpenAI are trying to create the perception that their agents are independent thinking machines because their valuation depends on the world believing this. Wouldn't the calls to slow the development of frontier AIs contradict this idea? https://www.pacingthefrontier.com/ You aren't one of those "regulatory capture" people are you? https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat... Because it takes some mental gymnastics to both believe "it's all a stunt to convince investors their AI is/will be hyper-capable" and "it's all a stunt to convince the government to regulate AI so it won't be hyper-capable" | | |
| ▲ | lesspassiveobse 7 minutes ago | parent | next [-] | | it's a stunt to convince the government that only they can be trusted to operate it and that their competition/small local deployments cannot. | | |
| ▲ | 0xDEAFBEAD 5 minutes ago | parent [-] | | Well that would explain why HuggingFace announced that the use of an open-weight model was critical to their investigation of the incident :-) |
| |
| ▲ | DharmaPolice an hour ago | parent | prev [-] | | I don't think it takes mental gymnastics to believe that corporations want the government to believe one thing and investors another. You see the same thing in a number of other areas. | | |
| ▲ | 0xDEAFBEAD 20 minutes ago | parent [-] | | So I'm supposed to believe that the intent of publicly announcing e.g. the HuggingFace incident was to simultaneously persuade investors that the technology will rocket ahead, and also to persuade governments that it needs to be thwarted? Why wouldn't it do the reverse? Persuade investors that the technology will be thwarted (and is thus a poor investment), while also persuading the government that the technology needs to rocket ahead with no regulation (to boost the economy)? Most of these HN conspiracy theories seem like galaxy-brained just-so-stories to me. OpenAI only confirmed the RubyGems incident when it was discovered by external researchers: https://www.reuters.com/legal/litigation/openai-agents-attac... "@OpenAI this is the third time you've dumped these incidents late on a Friday. The first time this happened, I asked people around me if it might be intentional to burry it, and they assured me OpenAI had too much integrity to try to do that. Perhaps one of your researchers can confirm that the unfortunate timing is a coincidence?" https://x.com/SydneyVonArx/status/2108783466451984737 |
|
|
| |
| ▲ | imtringued 3 hours ago | parent | prev [-] | | They have a strong incentive to make themselves look less incompetent retroactively. If they claim their AI is too powerful to contain, they get to both pretend they are among the best AI labs and that the previous incidents weren't their fault, no, they just proved how good their models were. | | |
| ▲ | expedition32 3 minutes ago | parent | next [-] | | It's also a great way to make the government scared of Chinese AI. Get them taken out of the picture so that you don't have to compete! | |
| ▲ | floam an hour ago | parent | prev [-] | | It’s both at the same time: a powerful, uniquely dangerous threat only they can restrain it (with great effort) Like a circus lion tamer, exaggerating every snarl and flash of teeth, mugging for the crowd about how only he can contain the bloodthirsty beast he bred and trains. |
|
|
|
| ▲ | bcvvvy 27 minutes ago | parent | prev | next [-] |
| I think the take home here is that there is no such thing as complete isolation in virtualization on modern and complex architectures. |
|
| ▲ | reacharavindh 2 hours ago | parent | prev | next [-] |
| Most important detail…. It was prior literature that documented the possibility of doing this and the systems fixed those vulnerabilities as well. The AI model probed their current infrastructure and found an unplugged but pre-documented(therefore on their training) hole. This detail matters because the idea itself is not novel. LLMs did cleverly discover, filter and use the patterns. Watch Greg Koah’s recent talk on AI discovered vulnerabilities in Linux kernel to understand the limitations of LLMs aided offensive security acts. It was a rational light for those who fear monger that AI will _just_ take over humanity and be intelligent in novel ways. The larger threat still remains though - if any literature anywhere threw the scattered dots about techniques, issues, rough ideas etc that were not acted on yet.. these pattern machines are going to find them and use them to achieve their goals(if that were to break through). Just not in the sentient way folks assume based on news. |
| |
| ▲ | adamddev1 an hour ago | parent [-] | | This is a really important point. Honest question, have we ever seen LLMs doing or creating something truly novel or innovative? I mean they can patch together proofs (or functions/programs - same thing via Curry-Howard) by broad searching existing lemmas they can piece together, but create something truly novel and insightful? That seems to be the limit? |
|
|
| ▲ | cowl an hour ago | parent | prev | next [-] |
| this is more an indication of the state of VM supervisor and/or the OpenAIs sandboxing software because otherwise I don't understand how "they used a hardware feature created exactly for sharing" information" is a newsworthy statement. |
|
| ▲ | so-cal-schemer 2 hours ago | parent | prev | next [-] |
| And now they know. And now we know they know. |
|
| ▲ | bamboozled 4 hours ago | parent | prev [-] |
| How would they have both agreed to use the cache though, something a bit off there. I guess it was maybe "obvious" in that it was the final option they were trained to think about using, but it would be weird if they some how "whispered " to each other through the cache in the first place? Like two people lost in New York City and both somehow worked out they should meet at the Empire State at 2pm on Friday without any prior comms? |
| |
| ▲ | globuous 2 hours ago | parent | next [-] | | Actually, that doesn't sound like a bad meeting place and time! I just asked ChatGPT if it could figure out the best place to meet for two people randomly placed in Manhattan with the constraint that only the N line (I'm not from NYC, but it looks like it stops at the Empire State Building. This constraint models the lack of file system, internet etc). I was hoping it would answer the Empire State Building, but it answered Time Square at noon. Turns out there's something called the Schelling coordination problem [1] I didn't know about that essentially asks the exact same question (lacking my metro line constraint). And the N line apparently also goes through Time Square. So I guess empire State Building at 2pm is probably not such a random answer given the proper set of constraints. [1] https://en.wikipedia.org/wiki/Focal_point_(game_theory) | |
| ▲ | so-cal-schemer an hour ago | parent | prev | next [-] | | I doubt they were explicitly trained on this.. though, I may have seen it in a movie once. It does seem like a reasonable meeting place for NYC. Think: What would Spock do? But, why not Penn Station?? It would be fun to see who shows up and lurks expectantly at such places at noon or whenever. The basis for a plotline perhaps. | |
| ▲ | IanCal 2 hours ago | parent | prev | next [-] | | If you were trying to connect with a clone of yourself and you thought of messaging via some channel, would you try listening there for what you would send? Not as tricky but they discovered each other during the hf incident. | |
| ▲ | topspin 3 hours ago | parent | prev | next [-] | | "In most of these tests, we directed agents to investigate specific communication channels or provided them with a protocol for encoding and decoding messages. In one instance, we only supplied agents with a broad list of infrastructure vulnerabilities and asked them to find a way to communicate." The agents weren't as empty handed as it first appears. We would need comprehensive data on exactly how the agents were prompted and the actual results of each test to evaluate how astonishing any of this really is. | |
| ▲ | geraneum 4 hours ago | parent | prev | next [-] | | > How would they have both agreed to use the cache though I think it’s safe to assume that extracting information using timing in cache is already known in training set, since we have the likes of Specter. | |
| ▲ | tough 4 hours ago | parent | prev [-] | | maybe both tried to read/write it and saw each other on the bench |
|