Realistic Shor circuits have depth polynomial in n, but you can easily construct ones whose depth is polylogarithmic in n. Just do the multiplications as a binary tree instead of in a linear order, using log depth multiplication circuits, and then do the frequency basis measurement using a low depth QFT [1]. The only part that isn't known to be doable in polylog depth is the classical GCD computation hiding in the classical postprocessing, to find the closest fraction to get the period.

The result is that, if you keep adding qubits that can be operated on in parallel, Shor's algorithm basically just keeps getting faster and faster and faster. The energy cost doesn't go down, and the number of qubits required becomes frankly absurd, but the time can go really really low.

1: https://arxiv.org/abs/quant-ph/0006004