> like every moment I spend with someone is just increasing or decreasing my score with them

This is more of a statement about the other person, especially if true, than the person trying to estimate the score, who is just trying to model their world as accurately as possible.

If you don't like it, the only thing you can do is try to be more complicated than a single score yourself. If it is in fact a good model of most human, then there is nothing you can do to change it, and being angry at the person who made you aware of the model doesn't help either.