I have to highly recommend the book 'Fundamentals of Musical Acoustics' for mathematical/physical explanations of sound and music. Bells having multiple resonant modes, tuning kettle drums, how string materials and plucking methods changes harmonic profiles, why having lots of high frequency harmonics sounds bright (and electric guitars, pianos optimise for this), why sounds without high frequency harmonics sound hollow (nylon strings), virtual pitch, acoustics of rooms, etc.

My impression is that most sounds have their effect on us due to psychological association with familiar sources of those sounds, and that melody and harmony are fundamentally more subjective - they depend on looser associations with motion, mood, the human voice, the changes and developments of sounds in our environment.

Are you talking about "Fundamentals of Musical Acoustics" by Arthur H. Benade? I'm interested in buying the book, but I'm not sure which one you mean. Do you perhaps even have the ISBN?