That's definitely a thing in portable speakers! In this case though the amplifier won't really know what it's powering, and different drivers have different sensitivity. The resulting volume will also be affected by the cabinet the driver sits in as well as the placement of the speaker in the room.

I wanted to hack something around this, since my amp, speakers, speaker placement, room and listening position are constant. But I have no idea how to actually go about this.

I can fetch the pre-amp level, but the actual sound level will be dependent on the source's actual volume which isn't constant (see: loudness wars). I could react according to a "measured" level, but how should I deal with a quiet portion (think classical music)? The closest I came was to use replaygain, but then that won't work with spotify...

I wonder if anybody came up with a solution to this.