But why not? If we look past the trappings of "we hate corporations", why not treat it as a friend? Let's say you acquire a free-trade organic GPU and run an ethically trained LLM on it. Why is an expensive funny-shaped rock not allowed to become a friend when a stuffed animal can?

The stuffed animal has only a sentimental value and it will not have a statistics based and geopolitically biased opinion that it shares with you and influences your decisions. If you want to see how bad a chatbot can be as a friend, see the recent case when it has driven a poor mentally vulnerable minor to suicide

There is a new term, AI Psychosis.

AI chatbots are not humans, they don't have ethics, they can't be held responsible, they are the product of complex mathematics.

It really takes the bad parts from social media to the next level.