There is a ton of questions to ask regarding their removal of cybersecurity data.

First, how much “cybersecurity data” is just standard software knowledge that one can derive cyber security exploits from?

Did they also get rid of all references to hacking in general?

Are they implying that all the recent exploits are just things that were already in the data?

For a model it doesn’t seem that far fetched to make the leap from “I have the knowledge of what hacking is” and “I know how computer systems work” to just being able to exploit stuff anyways.

▲

ByteAtATime 3 hours ago | parent | next [-]

I have essentially no knowledge in this field, but another concern may be that if the model has minimal cybersecurity knowledge, it may also implement insecure systems as a side effect.

	▲	zamalek an hour ago \| parent [-]
		My initial thoughts too, it's going to write insecure software. I recall early LLMs having problems with trivial stuff like SQL injection. Definitely going to give this one a skip.

▲

modeless 5 hours ago | parent | prev | next [-]

If you're an amazing, top 1% software engineer, and you switch jobs to become a penetration tester, you're not going to be top 1% day one. There's knowledge specific to penetration testing you'll need to research and absorb and practice before you're good.

For the models, it's always day one. They don't learn from experience. Whatever they know after training is all they'll ever know. So if you don't teach them penetration testing in training, they won't know it. Maybe they could theoretically derive everything from first principles, and so could a top 1% software engineer, but that's a very slow and inefficient way to do things.

▲

zozbot234 6 hours ago | parent | prev [-]

> For a model it doesn’t seem that far fetched to make the leap from “I have the knowledge of what hacking is” and “I know how computer systems work” to just being able to exploit stuff anyways.

You'd be surprised. Cyber offense is a very narrow domain, especially given the extensive mitigations that are being used in modern systems. Leaving it out of post-training tasks unless you're specifically interested in it is the no-brainer approach.

▲

jamienk 5 hours ago | parent | next [-]

Will there be a book of secret knowledge that only the initiated will be allowed to see? And they will all swear an oath to never input it into an AI system?

And then we'd also do that for all kinds of potential terrorism, sex-crimes, political subversion, and evil advocacy?

It would be the "restricted section" like in the Hogwarts library?

	▲	zozbot234 5 hours ago \| parent [-]
		Post-training is not about "secret knowledge": it's literally about training the model in a RL environment on very specific tasks (including, potentially, cyber exploitation). It's just as compute-intensive as pre-training these days, if not more (given that several checkpoints of the same pre-trained model are being released, showing sizeable gains in real-world capabilities for non-trivial productivity tasks).

▲

besterman23 6 hours ago | parent | prev [-]

Perhaps the gap is just too wide, but to use a more simple analogy. I see this like someone knowing everything about biology, simple mechanics, metallurgy, and human history but never hearing about an axe then assuming that person would never come up with the concept, use case of, or means to create an axe.