Also for some reason, most CI runners seem to cache nothing except for that minor thing that you really don't want cached.

This is exactly it - you can cache all the wrong things easily, cache only the code you wanted changed, or cache nothing but one small critical file nobody knows about.

No wonder many just turn caching entirely off at some point and never turn it back on.