I think your analogy is good, but misses the mark by millimeters. Optimizing is easy when you have a giant directory in the root of the drive named "delete this later". It's difficult when you have hundreds of thousands of directories, each with 100 files, of which you actually only need 75-85. You can't just delete any whole directory, because then things break; you have to go through each one and evaluate the purpose of each file, which is an expense in itself.

Fair point, although I get the feeling you may underestimate the state of disorganized chaos that my filesystems sometimes become.