I think the point is even stronger. alignment is really not very well defined, nor are the mechanisms used to provide it. from what I understand it's more of a statistically imposed bias than an impenetrable gate.

personally I would like to flip this around, all of this is a consequence of some really extreme notions about investment. its a tail-wagging-the-dog. the investment community is apparently in love with the idea of an AI-scale vehicle in and of itself. given the amount of power they have, this is funneling a massive amount of resources into something that while really very interesting, probably would never be able to realize proportionate gains without taking down the system that built it. what's the societal control on that.

▲

3 hours ago | parent | next [-]

[deleted]

▲

esafak 4 hours ago | parent | prev [-]

Probabilistic safety is too low a bar. We need to project the model outputs onto a safe subspace; lobotomize them, if you will. It may make them dumber, but that's a fine price to pay.

I don't work in this space so I don't know the latest, but here's an example: Provably safe systems: the only path to controllable AGI (https://news.ycombinator.com/item?id=37619285)

	▲	convolvatron 2 hours ago \| parent [-]
		I don't think the neuro-symbolic model is going to dumb anything down at all. Cyc didn't end up producing anything useful, but the only reason an llm is able to do math at all is that it can keep throwing stuff at lean all day. personally I think that that synthesis will turn out to be more powerful than just llm with constraints, but I don't really work work in the field either.