That has not been my experience. I develop on Windows and need to compile for Linux. After spending several hours trying to get cross-compilation working, I gave up and do it via WSL now.

I switched from Go and I feel like Go was much better at this than Rust.

(I tried “cross” but it was very slow and I found it faster to rsync the files inside the container and then run the build scripts)

I'd bet the difference is that Go has a default assumption that everything is reimplemented in Go and calling C is awkward and slow, meanwhile lots of low-level Rust libraries are actually just type-safety wrappers over C libraries.