|
| ▲ | InsideOutSanta 5 hours ago | parent | next [-] |
| > Seeing as no one besides blackhats want frontier cybercrime capability to be open source Sure. The problem is that there is no practical difference between "cybercrime capability" and "cybercrime defense capability." All of my PRs go through an adversarial security review by GLM-5.3, because it's the only smart-enough model with a good understanding of cybersecurity issues that won't regularly refuse the job. Removing that ability from frontier models helps nobody except cyber criminals. |
| |
| ▲ | stalfie 2 hours ago | parent [-] | | I'm not sure I follow. Let's say that GLM remains the frontier of open source cyber capabilities because Kimi is nerfed. Your situation, and the situation of your attackers, remain exactly the same. You run GLM to audit your code, they do the same looking for exploits. Let us say that un-nerfed Kimi was released, and it turns out to be at the absolute frontier of cyber capabilities, Mythos 5.5 level, a step ahead of anything else. What now? Well now your situation is exactly the same as before, just that auditing and hacking got more expensive. You run a Kimi security audit, they do the same to try to pwn you. However, now the cabal of Linux kernel defenders with secret access to Mythos 5.5 all of a sudden have no advantage, where before they used to have one. And even worse now they are in a literal race to be the first ones to find every bug that Kimi is capable of finding, and patching it yesterday. So for you, this makes no difference in practice, your capabilities are not elevated, and neither are those of the attackers. The only result of the other scenario is an overall defender disadvantage. |
|
|
| ▲ | ApolloFortyNine 6 hours ago | parent | prev | next [-] |
| The problem is, as anyone trying to address security concerns in claude can attest to, the guardrails trigger even when trying to address real bugs. Good cyber defense requires an understanding of the offense. How you avoid the "fix my code" attack, the code being someone else's code, I do not know. But nerfing the model doesn't seem like the best path forward either. |
| |
| ▲ | zozbot234 6 hours ago | parent | next [-] | | This is not about guardrails, it's just not post-training the model on tasks that specifically relate to cyber-offense. It's a very lightweight intervention, and it makes the model less of a target for bad actors. | |
| ▲ | tough 5 hours ago | parent | prev [-] | | claude code "auto-mode" uses a smaller sonnet-like model for classifying, which means it's basically dumb, and the harness biases it into denials |
|
|
| ▲ | tkz1312 4 hours ago | parent | prev | next [-] |
| I am a defender, and I really really want to have access to high quality models that are good at finding vulns and hacking. Leaving them in the hands only of government agencies and orgs with enough resources to do extensive training / fine tuning seems like the absolute worst possible case for the future security of global software systems. |
| |
| ▲ | stalfie 3 hours ago | parent [-] | | Well, and I am genuinely curious here, wouldn't defender and attacker capabilities being accessible equally by both parties be objectively worse than having a limited amount of privileged defenders? Of course, you could have every piece of code go through an adversarial audit before it's pushed, but in an equal world you would have to always devote as much or more resources to this as your attackers. And historically speaking people just don't do this in practice. In the scenario where only "verified" defenders get access to the best cyber capabilities, unverified plebian defenders and blackhats are on the same level exactly as described above, there's just also a few defenders that get to run things on godmode. So to me, it seems that the only result of frontier cyber capabilities is removing a defender advantage. Although that advantage is not for everyone, it can only exists as long as capabilities are unequally distributed. Am I missing something? | | |
| ▲ | Avamander 2 hours ago | parent | next [-] | | > Well, and I am genuinely curious here, wouldn't defender and attacker capabilities being accessible equally by both parties be objectively worse than having a limited amount of privileged defenders? Well, not really. Because those privileged defenders will never be able to predict what needs defending. So we have a few privileged "defended" places and the rest is left for privileged attackers to plunder. | | |
| ▲ | stalfie 2 hours ago | parent [-] | | Privileged attackers?? So that would be career cybersecurity experts with Project Glasswing/Daybreak/Googlewhatever access, using it to perform actual attacks? Because that's what privileged means in this context. Access to frontier models not available to the public. | | |
| ▲ | Avamander 2 hours ago | parent [-] | | > So that would be career cybersecurity experts with Project Glasswing/Daybreak/Googlewhatever access. Yes. You really think access those models will not be given for offensive operations in each country? Of course they will be. | | |
| ▲ | stalfie 2 hours ago | parent [-] | | Well, seeing as Claude literally snitches on everyday people planning murder and Houthis building rockets I would assume that every user in project Glasswing is heavily and very effectively monitored. I have to admit that I genuinely do not think that what you're hypothesizing is happening right now. I would bet good money that the amount of people with Glasswing access doing actual Blackhat work is currently zero. Perhaps in the future this will change, but even if that is the case, that's still a win for the defenders. Having a time advantage, even for a few months, is still a win. Edit: Actually, the CIA/FBI/NSA/Etc.. does probably have access and are probably trying to hack Iran and so forth, so my comment above is probably technically wrong in that respect. | | |
| ▲ | Avamander an hour ago | parent [-] | | > I would bet good money that the amount of people with Glasswing access doing actual Blackhat work is currently zero. I think it's not "blackhat" in that sense because it's officially sanctioned. Same goes for equivalent models overseas. The only thing these embargos achieve is giving massive nations a huge advantage, but leaves the average country and person exposed. | | |
| ▲ | stalfie an hour ago | parent [-] | | I edited my comment above before realizing this is what you meant. And its a valid point I concede. Still, I disagree that advantaging this activity is the only thing these embargoes achieve. I would argue that they legitimately also give the Linux kernel, cURL and so on an advantage. And as long as that is true, this still makes it an overall win in my books, even if it also enables officially sanctioned offensive activities. I'm not a US citizen, but I would still rather get pwned by the CIA than both the CIA and Shinyhunters. |
|
|
|
|
| |
| ▲ | tkz1312 2 hours ago | parent | prev [-] | | Because the people I want to defend against includes those with enough resources to train and fine tune models with strong offensive capabilities themselves. | | |
| ▲ | stalfie 2 hours ago | parent [-] | | Ah, now that's an interesting point. However, wouldn't those groups just fine tune Kimi K3.1 pro no matter if it's nerfed or not, leaving you in the same position as before? That is, unless the nerf is easier to correct as compared to "upgrading" the models offensive capabilities. So a scenario where it is easy to "restore" offensive capabilities, but hard to "enhance" them beyond the labs best attempt. So is that the world we're living in? |
|
|
|
|
| ▲ | ben_plotke 5 hours ago | parent | prev | next [-] |
| > Seeing as no one besides blackhats want frontier cybercrime capability to be open source I'm not interested in black hat anything, but I am very invested in making sure the products I work on are not compromised. Given that state agencies have "frontier cybercrime capability", I would like to have the same. |
| |
| ▲ | stalfie 2 hours ago | parent [-] | | I wrote a few lengthier comments on some other similar replies. Summarized my argument is that it seems to me to be strictly better that select defenders have an advantage, rather than everyone being equalized. If you're not one of the privileged few that might be frustrating, but open source models upgrade the attack and defense by the same amount, so open source cyber capabilities being at the frontier only results in a defender disadvantage. Replying here also as I'm curious about the counterargument. |
|
|
| ▲ | LtdJorge 5 hours ago | parent | prev | next [-] |
| It would also be very sensible to offer a BlackHat Pro version to national officials, for them. |
|
| ▲ | CBLT 6 hours ago | parent | prev | next [-] |
| Please don't change "cybersecurity" to "cybercrime" from one sentence to the next as a rhetorical flourish. It comes off as disingenuous and makes it harder to get your real point across. |
| |
| ▲ | stalfie 2 hours ago | parent | next [-] | | Noted. I also did it intentionally to sound a bit spicy, which seemed like a good idea at the time. So I am guilty, and I regret my actions. | |
| ▲ | zozbot234 5 hours ago | parent | prev [-] | | "Cybersecurity" as a general term often includes offensive "cyber" which is what is being talked about here. | | |
| ▲ | bbg2401 3 hours ago | parent [-] | | But the person being addressed specifically used the term cybercrime. |
|
|
|
| ▲ | verdverm 3 hours ago | parent | prev [-] |
| Black Forest Labs did similar with vision capabilities https://bfl.ai/blog/capable-open-and-safe-combating-ai-misus... |