Sure, that's what the chrony closed loop is already doing (the estimated residual frequency is pretty linear with temperature), but no matter how robust your closed loop is, it's strictly better to not have disturbances in the first place.