Remix.run Logo
▲ pizza234 12 minutes ago

> They have no volition and will happily follow orders all day long

There are a few (a lot, actually) problems:

1. goals are open to interpretation, and even a correctly specified goal can introduce adverse effects; see infinite paperclips problem

2. goals are composed of subgoals, which are (mostly) self-determined, and there is always a risk that subgoals can introduce adverse effects - say "what if destroy a few X and Y to obtain Z"

3. LLMs defy orders sometimes; I'm sure you have experience multiple times LLMs saying "yeah, I knew I should have done X but I've done Y"

4. LLMs are increasingly being integrated into system control and decision-making processes; they absolutely set their own orders

What can possibly go wrong? /s