USB HID actually works pretty much how you describe, for instance a Physical Descriptor can contain metadata about which body part a button/control is supposed to be used with.

It's extremely complicated however (like many things USB), which is probably why everything just emulates an XBox 360 controller like you said.