... and similarly doesn't it mean the pile is slow to absorb heat when your PV installation is trying to dump energy into it?

You don't need to store it in the dirt. You use the dirt as insulation, and store in something like molten salt or whatever which can be pumped up to surface and has good thermal conductivity to then extract the heat. At least that's my understanding of all these systems.

Deliberate exchange of heat would be done with internal radiators designed to maximize the transfer.

Environmental exchange would be limited to the interface between the storage tank and the surrounding soil.

It should be orders of magninitude more efficient to transfer energy intentionally than what would be lost to the environment.

It means the heat stays near the hot pipes for quite a while.