Very cool. I don't do alignment research but I have to imagine we do need to train models to be ethical. It's not enough to guarantee safety though, especially with LLMs. For one thing, the fact they can hallucinate means if you ask an LLM something enough times, you will get the unlikely result some percentage of the time. So that unlikely result has to be one you can deal with.

Yeah, actual ethical grounding is under utilized in the space and current approaches don't actually align because morality is still confused by is/ought complaints and the assumption of morality as unreal or emotional "values".

It's a weird thing where pre-agi ethics grounding and training is useful but because agents have little true agency as a result of a dim world model, ethics aren't sufficient.

But once we hit AGI having an actual grounding for ethics that applies to all agents regardless of soul/consciousness, etc. will become all that matters.