They really don't, and if they did then would it be so bad if people who didn't "need" them took them?

Obviously if there's safety issues but for stimulants unsafe doses will 100% always decrease performance, because they'll affect sleep.