1) It takes a long time to change cryptography standards, and cryptography also generally needs to provide some forward secrecy: i.e. you would like that something you encrypt today is safe for some time afterwards. This means the timelines cryptographers care about when advocating for a change are long, often more than a decade.

2) By most accounts, quantum computing scaling is limited by various physical effects causing noise in the circuits that prevent making larger circuits from the smaller ones that exist. If this was the end of it, then you would expect scaling to be very slow and probably infeasible. But there is also a process of quantum error correction, which means that once you can build a large enough and reliable circuit to implement it, you can scale very easily. This makes quantum computing scaling very nonlinear: it is expected that scaling will suddenly become a lot easier once this threshold is reached, and it sounds like the state of the art is getting close to that threshold (you can of course find people who are skeptical of this claim: from believing that the timelines are optimistic to doubt that the physics works at all).

3) cryptographers are also likely to err on the side of caution: the effects of widely-used encryption being broken are very bad, so it's best to assume that QC will accelerate quickly and that adoption of post-QC cryptography will be slow.