> The former, because it's anthropomorphizing a model.

Not really. The input and output is already natural language. That is already "anthropomorphizing".

That is, if this is the bar for anthropomorphization its already happened.

Telling the model to "believe in itself" is just stochastic manipulation that has shown enough reliability to be a recipe to make it keep going.

It's only actually anthropomorphizing if you forget it's a trick and think it's a real person.

There is nothing distasteful about it. If people get confused that's on them. They wouldn't be very useful if you couldn't just talk to them. That's kind of the whole point. Otherwise you can just go back to coding by hand. Telling it to believe itself is just input that happens to work. This probably tells us more about human nature than you realize given the corpus on which it is trained. It obviously doesn't mean anyone actually thinks it's a person.

>There is nothing distasteful about it.

It's obvious that you don't get it but I will try my best to explain why so at least you can form an idea about how others feel.

It's about what makes humans unique. The LLM does not experience reality, it just merely pretends it does, and even that, it does in a shitty way. I think disgusting is a very adequate adjective. The reason why it is disgusting is because you are devaluing a divine experience to the realm of the common and the vulgar, a cheap substitute being valued as equal (or even on the same scale) as the most important experience we could go through.

To give you an example that might land in a more familiar context, think of that one guy who takes his plastic doll everywhere and pretends it's his wife and gets upset when others don't acknowledge "her" as a person.