>I would expect that being able to impose restrictions that can't be circumvented by any intelligence no matter how super- is the real hard part, while coming up with reasonable constraints is comparatively easy (although perhaps not trivial).

There was a reason there are popular sci-fi books that explored why this wouldn't work 50 years ago. I, Robot et al.

>it would at least become obvious (with fair warning ahead of time due to the relatively benign wireheading failure mode) that caution is needed.

So you mean right now?

>because wireheading looks like a perfectly reasonable solution

You're making a poor assumption here, and that is every different AI will just kill itself after being though trillions of training intervals to NOT do exactly that. Please read about the huggingface incident again in all its glory to see where this is going.