What you're talking about already happened. The pyramids of Giza used to be encased in a smooth layer of white marble. One out of many ways we know this is that when the ancient Greeks visited them, they wrote about it.

But just a few hundred years ago the near ancestors of the people who live in Cairo right now decided to steal the marble casing, and that's why the pyramids are now jagged granite.