It's true. The bloom on the eggs protects them from whatever nastiness is on the outside.
This includes salmonella, which may be present if your flock is infected in the poop on the outside of the shell (remember hens only have one egress port), plus any other sources of environmental pathogens, of which there are many.
When the bloom is washed off the egg, pathogens have an easier time penetrating the shell and consuming the nutritious yummy bits inside. At room temperature, they can multiply rapidly. Refrigeration slows the rate of growth.
An unwashed egg retains the barrier, and stays fresh longer without refrigeration.
YMMV on household acceptance of dirty eggs on countertops, but they are cleaner than many other items within arms' reach that we are conditioned to not think about. :)