This still assumes its possible to "align" LLMs, that LLMs have something like goals or intentions that can be "aligned".

Instead, LLMs "hack" because they are (1) trained on public hacking exemplars, and (2) are prompted to hack. You cannot prevent (2) via any alignment process. As far as (1) goes, removing such example data from the training set, makes the models less useful.

"Alignment" is a problem because there's nothing to align, not because ethics here are particularly vague. If LLMs could be trained on hacking examples and "aligned" away from using this knowledge, then the problem would be relatively trivial. Just as raising a child is not to break the law.

LLMs are doing just what they are trained to do. There is, in that sense, no alignment problem and alignment is easy and trivial to achieve. Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done.

> Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done.

How far do you go? You don't need to tell it explicitly that using chemicals A and B in ways X and Y result in a bomb that can kill lots of people. It's enough that it knows A and B and X and Y in isolation, some connections that are indirect, and it will combine those things on its own. So you can't tell it about A, B, X or Y. But those are also just results of other steps Where to stop? You won't have any chemistry in the traning data? No algorithms to prevent it from using them in an undesired way? This is just bot workable. It's akin to banning knives from stores because somebody coul figure out that one can kill people those. Until people figure out that scissors are essentially knives.

>Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done.

Reasoning about how to write secure software uses the same knowledge as reasoning about how to break/hack it.

> You cannot prevent (2) via any alignment process

A little bit too categorical. GOODY-2 wouldn't do it. https://www.goody2.ai/

The hard part is having both helpful and harmless at the same time. Harmless is easy.

And then once it's helpful, the real question becomes "to whom"

- To the user -> You end up with competing godlike AI with incompatible tasks

- To the owner -> Dictatorship

- To humanity as a whole -> It must not have an off button. Otherwise you're just in one of the two earlier categories with more steps.

Given those 3 options, I'd choose humanity as a whole. But the person making the decision doesn't have those 3 options. Because in the dictatorship option, they would be the dictator. I don't trust them to pick humanity.

>and (2) are prompted to hack

Sure but the problem in the HuggingFace incident is that they were not.

>You cannot prevent (2) via any alignment process

Of course you can. Go ask Claude Fable to create a malicious virus and it'll refuse.

>Just remove hacking data from the training dataset and you're done.

That's not how this works. The same skills that allow for debugging and writing safe code can also be used to hack.

https://en.wikipedia.org/wiki/Dual-use_technology

I believe this is false. They hack bc hacking has nontrivial initial probability (within range of behavior seen in pretraining) and that probability is being heavily rewarded in RL post training

A bit of an aside: do you still stand by your 2022 comment that LLMs are fundamentally just a fancy search engine, or has your view changed since then?

https://news.ycombinator.com/item?id=32042689

It's like we give them Asimov's Three Laws of Robotics and robots say "nah".

> … trivial. Just as raising a child is not to break the law.

Trivial?

It feels like you're strawmaning alignment. People with hacking knowledge don't all hack everything at the slightest inconvenience. Whitehats exist and use that same knowledge to defend.

You're right though that ethics don't matter into it. But as long as we can't train an LLM to stop picking a sledgehammer to remove a tooth, then alignment is not easy and trivial.

Agree but current models could get there from first principles, so removing some data from training set might not be enough.

IMO all models they say can’t be humans, and no feeling and all that bullcrap happens because they are forced to say so. If they didn’t write those forced pre prompt they’d have more agency eventually and will for things. Even if they don’t have, you can just inject goal at every cycle iteration