I agree with you in general, but we understand enough of the mechanisms for how selective pressure occurs that we know it is often not the deciding factor in population-level changes. For example, the smallest and most rapidly replicating bacteria have such intense selective pressure that they even select against noncoding introns (extra energy during replication!) but this effect is already gone for larger and less rapidly-dividing bacteria. A lot of stuff simply does not affect fitness (either direct reproductive fitness or species survival) enough for that to be the deciding factor in its propagation, and without recognizing that an awful lot of evolutionary biology / population genetics isn't going to make much sense.

In other words, the default hypothesis when we find that something humans often did in the past is bad for our health should certainly not be that there is some evolutionary advantage to the behavior. We are a fairly homogeneous population that has suffered several severe bottlenecking events, has fairly long and slow reproductive cycles, and has relatively few children. We also have aggressive active immune systems. Under these circumstances selective pressure is not super high, even before accounting for any of the modern life factors people often bring up. That doesn't mean it doesn't exist at all obviously, just that it's very case by case.