does turbopack make such a difference on next.js sites?

Yes, because it is yet another Typescript and Webpack like compiler written in Rust.

Note how many HNers are making the same remark.