| |
| ▲ | wand3r 2 hours ago | parent | next [-] | | I genuinely would like to know your opinion on what the next steps would be then? If every single exploit was available tomorrow accompanied by a working exploit to operationalize it, or combine it in a chain to compromise any software that has such a vulnerability, would you post it on the internet? | | |
| ▲ | tptacek 2 hours ago | parent | next [-] | | You're asking if I support full open disclosure? I spent most of my career as a vulnerability researcher. I believe in disclosure. | | |
| ▲ | wand3r 43 minutes ago | parent [-] | | I know you a bit by reputation so that was my assumption. I assume you mean responsible disclosure? I thought your statement was implying otherwise, maybe the unsaid implication was against the "extortion" argument? In any case, what I want to say is this: All technology has tradeoffs but this has shown we are locked in a game theory mechanic not dissimilar to extortion AND the security implications are part of that. - We have to build AI or someone else will. - You need our tools to protect yourself from our tools or similar ones - You need to use AI or someone using AI will take your job. Many examples but it is very "i created the disease to sell you the cure". If this were any other technology, we would expect more reasonable mitigations, extraneous factors like software bugs existing aside. It speaks to a fundamental lack of control on the part of the labs that we take for granted: they simply don't have control of the PRODUCT, much less how it is even used. | | |
| ▲ | tptacek 41 minutes ago | parent [-] | | Absolutely not. "Responsible disclosure" is a coercive Orwellian term coined by vendors, most importantly Microsoft, in the early aughts in order to set a norm that vulnerability researchers should work on vendor timelines and with vendor disclosure constraints. Finding vulnerabilities that already exist in code can't be "creating the disease", for obvious reasons. | | |
| ▲ | wand3r 20 minutes ago | parent [-] | | Ok. I was sympathetic to that historically for all the reasons you stated. If you put the effort in, why are you constrained. I think we are in a different world now. So what would you do in my thought experiment? You are the QA head and the new model could literally find and exploit ANY bug? Would you release the model? Genuinely curious. Its not totally science fiction. I think there is a difference in a couple smart people punishing 1 sloppy company that underinvested in security. The internet is a stack of protocols, hardware, infrastructure, applications etc. It could be the case that almost everyone has capabilities on par with yours. Great you say, security will improve! However, there will be a meltdown in the middle where potentially digital currency is exploited, exchanges, banks, utilities. It isn't just a team working for 3 months, finding some issues in a company. Or is this way overblown? |
|
|
| |
| ▲ | ericst an hour ago | parent | prev [-] | | I cannot help to think that the idea of go fast and break things had something to do with the current state of affairs. I would say that the next step should be slow and consolidate, but it seems like we are not heading in that direction. At this point I don't know what will be worse for the global economy the AI Bubble bursting, or the gigantic crash all this vulns might bring if they force the stop because we went too far and everything came crumbling down... |
| |
| ▲ | 112233 2 hours ago | parent | prev | next [-] | | Ah yes. Everyone eventually dies, we just shorthen the wait. This is like telling that bank vaults always were breachable, and giving out explosives to everyone just made them get breached sooner. (edit: fact of existance of a vulnerability is not a binary yes/no. the amount of effort needed to exploit even known vulnerabilities is usually staggering, requiring months from very capable individuals with very specific skillset.) | | |
| ▲ | tptacek 2 hours ago | parent | next [-] | | In your analogy, one phenomenon concerns a living organism and the process of senescence, and the other a piece of information. I don't see how you connect the two. | | |
| ▲ | 112233 2 hours ago | parent [-] | | Both my analogies use hyperbola to comically exaggerate the claimed lack of any difference. just as claiming that advancing a demise of a human makes no difference because it will eventually happen, or claiming that breaching a bank in practice is not a big deal because it was possible in theory, so also claiming that theoretic possibility of a vulnerability being exploited is more or less the same as many of them being exploited at the same time. The covid was not a lot deadlier and not a lot more contagious than normal seasonal flu. That "not a lot" overloaded hospitals and caused global shutdown. The increase in rate at which software is being breached right now in in no way "not a lot". Devs were overloaded even without this deluge. We can tank one such php exploit. Then another one. But for how long? If all your smartcards, secure elements, cloud providers, wireless protocols, drivers, government registries etc etc will be having weekly breaches of this magnitude, won't the effect be systemic? | | |
| ▲ | tptacek 2 hours ago | parent [-] | | None of these analogies are apposite. They don't respond to the observation I made: the vulnerabilities we're discussing exist regardless of what OpenAI does. | | |
| ▲ | 112233 an hour ago | parent [-] | | The fact that vulnerabilities exist is not being disputed, and is not causing any controversy. The fact that vulnerabilities are getting exploited does. Even having AI that would simply find all existing vulnerabilities and list them would not be as destructive as current situation. The same AI that finds the previously unknown vulnerability also finds the way to exploit it (NOT THE SAME THING), writes actually usable exploit (also not the same thing), then orchestrates and oversees the mass campaign. THAT is what makes current situation unsustainable. Let's all recall old times when vulnerabilities of this magnitude were discovered - windows defender had some nice ones, linux kernel did, a lot of software did.
They got discovered, communicated to vendor, patches were made, people updated. Now? You have HOURS from discovery to full-out campaign. You published a patch? HOURS before it is disassembled, all fixed bugs PoC-ed and full out campaign started. You cannot update anything meaningfully in that window of time. | | |
| ▲ | tptacek an hour ago | parent [-] | | Your premise, whether you mean it to be or not, is that if OpenAI wasn't working with researchers to publish vulnerabilities, they wouldn't be exploited. That's false. | | |
| ▲ | 112233 an hour ago | parent [-] | | (where in the article did you read about OpenAI working with researchers to publish vulnerabilities?) Yes, exactly, that is my premise (minus the "working with researchers" bit). The current rate of exploitation is direct consequence of actions of Anthropic, OpenAI, Alibaba and whoever else is making these things. How can I know my false premise is true? By looking at the situation just a year ago, that is how! You are focused on the binary: vulnerability exists or it does not. It can be exploited or it can not. That is not what is causing the issues! |
|
|
|
|
| |
| ▲ | injidup 2 hours ago | parent | prev [-] | | Liam Neeson? |
| |
| ▲ | JTbane 2 hours ago | parent | prev | next [-] | | Honestly, it's not a terrible idea to start asking people to run AI agents as a red team to find vulnerabilities. | |
| ▲ | avaer 2 hours ago | parent | prev | next [-] | | > The vulnerabilities exist whether or not the AI does. No. Even if we ignore the people newly enabled to slop out vulnerable software without having a clue about security, the AI powered supply chain hacks inserting vulnerabilities, the influx of AI contributions that drown out improvements in FOSS security, and the industry pressure to ship faster while reducing quality or be left behind -- all because of AI -- security through obscurity has historically been the most effective security. I don't think your argument holds under theory or (especially) practice. | | |
| ▲ | lioeters an hour ago | parent | next [-] | | > security through obscurity has historically been the most effective Security through mathematical impossibility is the most effective. AI-assisted brute-force vulnerability search is revealing that what we considered obscure a year ago is no longer hidden and was never secure. | | |
| ▲ | tptacek an hour ago | parent [-] | | I think the even more important observation here is that the cat is already out of the bag on this approach to vulnerability research; it can be done effectively with open-weight models today. | | |
| ▲ | lioeters an hour ago | parent [-] | | I see it's an arms race between white and black hats using this new tool to expose vast new areas of vulnerabilities and fixing them to improve security as fast as possible. It's always been a cat-and-mouse chase, or maybe more like a whack-a-mole, but now accelerated with super-human ability on both sides. Fast-forward a few years of this, I wonder whether they will have found all the low-hanging fruits (from AI perspective) and the situation will settle down; or it will keep getting worse - with AI development never reaching a plateau - until we admit that all existing software and hardware are hopelessly vulnerable and likely compromised already. | | |
| ▲ | tptacek an hour ago | parent [-] | | The capabilities open-weights models had months ago already made this a foregone conclusion. |
|
|
| |
| ▲ | tptacek an hour ago | parent | prev [-] | | The vulnerabilities we are seeing in the news are, as a rule, not things introduced by AI. That could change, but this rebuttal isn't on point currently. |
| |
| ▲ | pmlnr 2 hours ago | parent | prev [-] | | [flagged] | | |
| ▲ | tptacek 2 hours ago | parent [-] | | My comment is responsive to the article and to the parent comment. | | |
| ▲ | pmlnr 24 minutes ago | parent [-] | | Right.
The article basically says you need AI to fight AI, and it's barely about the vulnerabilities. So why responding only to the vulnerabilities? |
|
|
|
