It's really that simple.

I run a website that's primarily text-based. When I change the base template, I still check that it works without CSS. This just means semantic HTML.

That being said, CSS is rarely that large. Even after a few years of relative indulgence, the gzipped CSS for the whole website is still something like 20kb.