You're missing a critical piece of information. Human hearing (and vision) are logarithmic sensors.

Ears can register sounds from maybe 20-30 dB upwards of 120ish which isn't a factor of 4-6 in terms of power but rather a factor of 120-30=90 decibels or 9 bels or 10^9 or one billion.

Because your ears have absolutely enormous range you need the potentiometer (pot) to have a logarithmic taper to it. The amplifier has an essentially fixed amount of amplification so that's a fixed sound dB output. Your ears can hear a vast range. A linear pot essentially locks the entire output into the same 10 decibels as the amplifier maximum output through its linearity. Once you've turned it to 10% of the range it has precisely 10 decibels worth of range left. If you want to turn the volume down by 40 decibels you have to do that within the 0-10% part of the pot's range.

A logarithmic pot will give you maybe 40-60 decibels worth of adjustment by dividing things up differently. Every 20% of the range increases the output not by 20% but by a factor of 10 let's say. That gives you a pot with a range of 50 decibels which is enough that it roughly matches the absolutely miraculous range of the ear.