So? It would apply to sex chromosomes and only sex chromosomes, which is just one observed sex characteristic.

We are talking about sexual dimorphism and secondary sex characteristics.

Humans were understood to be sexually dimorphic before we discovered sex chromosomes in 1905, and we usually label our babies with a biological sex without the aid of consumer genetic testing.