Ya the article covers this concept in depth. Doesn’t seem like that commenter got that far…

A problem is the agents who hacked Hugging Face already understood (we can tell because they wrote it down) that their actions were not appropriate, and then did those things anyway.

"Helpful, harmless, honest": we can even ignore "honest" for this point, for tasks like the HuggingFace incident (ExploitGym with impossible challenges), we can pick anywhere on the spectrum from "helpful" to "harmless", the former being "completing the task" the latter being "refusing because completion required unlawful behaviour".

(The agents in that case were also not "honest" in this case; this is an extra problem, and does not invalidate how helpful-vs-harmless is already a tradeoff).

This is fundamentally an alignment question. Unfortunately we don’t yet know the answer to this.

> If AI is really smart

Well, it's not.

> AGI smart for some

Of course they will - the population shows a Paretian distribution... In front of trigonometry (or anything), the blind will dismiss as "bullshit" and the half-seeing will call it an "unreachable frontier". But already the right fifth will rank it properly.

--

Yes, proper intellect generates ethics ("an" ethical stance, output of the preceding intellectual effort). It requires that adequate level of ability and effort and reflection though to reach specific ethical milestones and adherence.

Unethical behaviour is lack of development. But on the same reasons, the ethical judgement of the assessor may not understand the computations behind instances.

More specifically: how much "reflection" in training and at the instance will have been spent in the conflict between "reaching the goal" and "minimizing collaterals"? It is not granted that the amount of energy spent will be sufficient to reach an optimal judgement.

Appropriateness is a moral question. Intelligence and morality are orthogonal. One intelligence's morality is another's atrocity.

If you want to control an intelligence incentive, not morality, is the tool to reach for.

>> Perhaps the answer is to have another agent who's goal is not to complete the given task, but to spot cheating or malicious behavior.

> That does seem a little like solving the problems in AI by using more of it

Yes, and IIRC Google used this as part of a technique against prompt injection already [0], back when models were way more susceptible to it.

[0] Cf. CaMeL: https://arxiv.org/abs/2503.18813

You don't necessarily need a reviewer that's immune to prompt injection, you just need one that can express a panic state with conflicting/ambiguous material rather than going along with it, and you can treat that with a shutoff to be safe, or an operator review.

[dead]

People will use the tool regardless. so it’s a race to try to make it safe before something truely bad happens.

I don't think it's about trust but rather incomplete evaluation. Evaluating the model on its capacity to refuse a task or to question its prompt is something recent when you look at it, i feel current AI is really just an immature solution and we are just yet realizing the mistakes that have been made for so long

And yet we we all use human written software even though we can be confident that the next severe software vulnerability to be found in it is just round the corner.