That sounds like the compilation is accidentally triggering this old frustration [0].
[0] https://github.com/python/cpython/issues/131189
Compilation was slow for me on macOS too.
We'll improve this very soon, right now the experience is less than ideal.
Ideally we would download the compiled artifacts instead of compiling as Simon commented... it will be a much better experience for everyone!