Mining is industrial chemistry. If you dig up a bunch of lead or arsenic that is mixed with some other metal you actually want, you have to put that arsenic or lead somewhere. You can covert it to different forms of lead or arsenic but they will always contain those elements. At which point, the only question is how much money you want to spend to convert those toxic waste products into a specific form that may be more manageable or slightly less toxic when you dump them. When we talk about "downsides" to mining, this is the elephant in the room because we can't make those elements not exist and mining will always produce them.

Most metals commonly occur together with specific other metals in nature. For example, it is rare to find silver and zinc without a lot of associated lead. You can't make that lead disappear and we still need silver and zinc.