> or even putting thermal mass around the existing oscillator

I was thinking along these lines as well. Put a metal block on the CPU and oscillator for thermal mass (not sure if separate blocks would be better). Ideally, with a large enough thermal capacity, the block should reach an average temperature and remain there.

Inertia is also good even if the temperature is not constant: clock drift can be measured and compensated. If the temperature rises slowly, the clock speed will increase slowly: the rate can be measured and compensated for. Jitter is the issue here, and thermal inertia should dampen it.

It may also be worth preventing convection from happening on the board. Putting the Pi in a wool sock may not be the best idea depending on its temperature, but an electrically insulating thermal conductor (or an electrical insulation layer + steel wool may do it).

Heatsinks may also be counter-productive (if they have a small thermal capacity), as their temperature depends on room temperature, which changes during the day.