I regularly tell new people at work to be extremely careful when making requests through the service desk — manned entirely by humans — because the experience is akin to making a wish from an evil genie.

You will get exactly what you asked for, not what you wanted… probably. (Random occurrences are always a possibility.)

E.g.: I may ask someone to submit a ticket to “extend my account expiry”.

They’ll submit: “Unlock Jiggawatts’ account”

The service desk will reset my password (and neglect to tell me), leaving my expired account locked out in multiple orthogonal ways.

That’s on a good day.

Last week they created Jiggawatts2.

The AIs have got to be better than this, surely!

I suspect they already are.

People are testing them with trick questions while the human examiner is on edge, aware of and looking for the twist.

Meanwhile ordinary people struggle with concepts like “forward my email verbatim instead of creatively rephrasing it to what you incorrectly though it must have really meant.”

There's a lot of overlap between the smartest bears and the dumbest humans. However, we would want our tools to be more useful than the dumbest humans...