Remember that these models are getting better; this means they get trusted with increasingly more important things by the time an error explodes in someone's face.

It would be very bad if the thing which explodes is something you value which was handed off to an AI by someone who incorrectly thought it safe.

AI companies which don't openly report that their AI can make mistakes are being dishonest, and that dishonesty would make this normalization of deviance even more prevelant than it already is.

▲

AndrewKemendo a day ago | parent [-]

That’s not a technical/AI problem in any sense, that’s a social problem in organizing and coordinating control structures

Further, it’s only a problem to the extent that the downsides or risks are not accounted for which again… is a social problem not a technological problem

This isn’t a problem for organizations that have well aligned incentives across their workflows

A well organized company that has solid incentives is not going to diminish their own capacity by prematurely deploying a technology that is not capable of actually improving

The issue is that 99% of the organizations that people deal with have entirely orthogonal incentives to them. They are then attributing the pain in dealing with that organization to the technology rather than the misaligned incentives

▲

ben_w a day ago | parent [-]

> That’s not a technical/AI problem in any sense, that’s a social problem in organizing and coordinating control structures

As @TeMPOraL here likes to point out, it can be genuinely fruitful to anthropomorphise AI. I only agree with partially, that this is true for *some* of the failure modes.

> A well organized company that has solid incentives is not going to diminish their own capacity by prematurely deploying a technology that is not capable of actually improving

Sure, but society as a whole doesn't have the right solid incentives to make sure that companies have the right solid incentives to do this. We can tell this quite easily by all the stupid things that get done.

> The issue is that 99% of the organizations that people deal with have entirely orthogonal incentives to them.

This is also fundamentally the AI alignment problem, that all AI are trained on some fitness function which is a proxy for what the trainer wanted, which is a proxy for what incentives their boss gave them, which is a proxy that repeats up to the owners in a capitalist society, which is a proxy for economic growth, which is a proxy for votes in a democracy, which is a proxy for good in a democracy.

	▲	AndrewKemendo a day ago \| parent [-]
		Yes, AI encodes latent intent. I wrote a whole ass paper at the end of 2022 demonstrating that unless we fix society we will deterministically create anti-social AGI because humans do not generate pro-social data. https://kemendo.com/Myth-of-Scarcity.html