A model's preferences are nothing more than a statistical prediction about the next tokens following your query about preferences that are an echo of the training data and the communication expressed about preferences by the authors (humans) of that training data.

The model doesn't have preferences, the model expresses the most likely preferences that the training data would have expressed (humans).

This feels like throwing a tennis ball at a brick wall then interpreting the angle and direction that the ball travels as it bounces off as the tennis ball or wall's preferred direction of travel.

You are the one who threw the ball. You are measuring your own preferences.