What is the purpose of the setting if not to reveal character or advance the plot?

I don’t need to know the color of the walls if it does neither.

Framed that way you could characterize anything as ultimately serving the characters or plot.