So if I run a test suite or compile my rust program in a sub agent I’m going to get cache misses? Boo.

Sub agents don't have much context and don't stay around for long, so misses in that case are trivial.