Now that I look at it, why is the arm64 that much smaller then x86_64?

36 vs 47MB

Just an educated guess, but perhaps there are optional dependencies missing in the arm64 version. Some programs are still broken or perform badly on arm64.