If rich people stopped going to Stanford, Stanford will loose its reputation in a few generations.

I think it works in the opposite direction. Rich parents basically buy admission for their mediocre offspring at a university made prestigious by the abundance of very intelligent but less wealthy students.