If the safest drug is also the one most prone to misuse, and misuse makes it unsafe, then it is no longer the safest drug. The reason society should be wary is precisely because people misuse it on a scale that vastly eclipses garden variety drugs like acetaminophen.

Yeah, "misuse" IS a side effect.