Or someone told them that folks in Egypt made a huge building out of stones. They started and gave up after a while. As good a story as any?

Stonehenge is older than the pyramids though.