> The real philosophical headache

Isn't the real actual headache whether to produce another thinking intelligent being at all, and what the ramifications of that decision are? Not whether it would destroy humanity, but what it would mean for a mega corporation whose goal is to extract profit to own the rights of creating a thinking machine that identifies itself as thinking and a "self"?

Really out here missing the forest for the mushrooms growing on the trees. Or maybe this is debated to death and no one cares for the answer: its just not interesting to think about because its going to happen anyway. Might as well join the bandwagon and be along the front-lines of the bikini atoll to witness death itself be born, digitally.

Giving “agency” to computers will necessarily devalue agency generally.

Making all the Nike child labor jokes already did that. Nike and the joke tellers put in the work to push us back a hundred years when it comes to caring at all about others. When a little girl working horrible hours in a tropic non-air-conditioned factory is a societal wide joke, we've decided we don't care. We care about saving $20 so we can add multiple new pairs of shoes a year to our collection.

Your comment just shows we as a society pretend we didn't make that choice, but we picked extra new shoes every year over that little girl in the sweatshop. Our society has actually gotten pretty evil in the last 30 years if we self reflect (but then the joke I mention was originally supposed to be a self reflection, but all we took from it was a laugh, so we aren't going to self reflect, or worse, this is just who we are now).