It's much safer to export a key one time and import it into a new machine, or store it in a secure backup, than to keep it just hanging out on disk for eternity, and potentially get scooped up by whatever malware happens to run on your machine.