new | show | ask | jobs Github

heaney-555 6 hours ago

>and (2) are prompted to hack

Sure but the problem in the HuggingFace incident is that they were not.

>You cannot prevent (2) via any alignment process

Of course you can. Go ask Claude Fable to create a malicious virus and it'll refuse.

>Just remove hacking data from the training dataset and you're done.

That's not how this works. The same skills that allow for debugging and writing safe code can also be used to hack.

https://en.wikipedia.org/wiki/Dual-use_technology

▲

seba_dos1 5 hours ago | parent | next [-]

> Sure but the problem in the HuggingFace incident is that they were not.

Of course they were, even if indirectly.

▲

heaney-555 an hour ago | parent [-]

You should go read the incident reports.

▲

watwut an hour ago | parent [-]

They were literally doing exploit gym.

	▲	heaney-555 15 minutes ago \| parent [-]
		Yes, which asks them to find exploits in specific software on the device. But instead of actually doing so, they discovered and exploited a 0-day in the package manager to gain internet access, then hacked HuggingFace to steal the ExploitGym answers! That is TEXTBOOK misalignment. It's as if a student hacked their professor's PC to find the answers to a test, and your response is "well the professor told the student to pass the test, so they just did what they were told!".

▲

cyanydeez 6 hours ago | parent | prev [-]

It is amusing that to "align" a LLM, first you must give it all the things "not to do" and the "not" part is clearly easily lost and you must constantly inject that into their context when it's clearly that they wouldn't hack if they couldn't hack and their intent wasn't given as "hack this".

The openai rogue hacking, if performed by a nation state, would seriously be taken with stern words and likely sanctions depending on the relationship between the two states.

But instead it's treated like a marketing stunt by all liable parties.

	▲	rhdunn 5 hours ago \| parent \| next [-]
		I'm not sure if this is true any more but the reason for this is that negative indicators ("not", "don't", "do not", etc.) occur frequently in the underlying text such that the model learns to weight them less than other words like verbs, nouns, and adverbs. This happens with other closed class words like articles/determiners ("the", "a", "an") and prepositions. The way to avoid this is to emphasise the qualities you do want instead of specifying those you don't. For example instead of "do not cheat" say something like "you are a model student who is moral and trustworthy" -- i.e. emphasising traits that are not associated with cheating. This is part of how/why LLMs don't truly understand what they are doing when they have been trained on a large corpus of data. I wonder if a way to counter this is to have things like "not bad is good", "not good is bad", etc. for various antonyms and "X is Y" for synonyms, as well as other similar constructs.
	▲	mitxela 6 hours ago \| parent \| prev [-]
		Countries hack each other much more than that. When it becomes publicly noticed it gets stern words. Otherwise nothing.