The difference with humans is that you can incentivize them to try and do the right thing via money. Humans are error-prone and slow, but they don't have an alignment problem, and the types of mistakes they'll make are pretty much understood. There are entire programming languages designed to minimize those types of mistakes.

The types of mistakes LLMs will make are failures of alignment, that is, basically unpredictable. They could fuck up anything. Where a human fucks up by introducing a crash, the LLM fucks up by just not solving the problem you asked it to solve.

It will happen less and less as they get better and better, but as the volume of software output keeps growing, it will happen with a consistent defect rate. And we won't be able to catch it prior to shipping, because humans won't be able to review all the code or verify all the tests.

I don't think anyone's started taking this seriously yet.

> The difference with humans is that you can incentivize them to try and do the right thing via money.

Gen Z has learned that usually, your dedication to work is not tied to the monetary reward.

> Where a human fucks up by introducing a crash, the LLM fucks up by just not solving the problem you asked it to solve.

Have you ever done a group work project at school?