It's the same logic as any offensive technology: maybe the world would be a better place if we never invented the technology, but we can't risk our enemies having it while we don't, and even if they never develop it maybe it'll help us, and we're the good guys.

Luckily, in this particular arms race, all we the public need to do is swap encryption algorithms, and there's no risk of ending global civilization if we mess up. So we get the best of both worlds: Quantum computing for civilian purposes (simulations and whatnot), while none of the terrifying surveillance capabilities. We just need to update a couple of libraries.