On my M1 Mac Pro I get 0.13s wall, so not much faster than your Mac. On my i9-9900K Linux box I get 0.04s. I would think my M1 single core performance would be on par, if not faster. Perhaps it has something to do with macOS and gatekeeper, as I notice I'm not getting as high of a CPU utilization.

    $ gtime /opt/homebrew/bin/emacs --batch --eval '(princ (format "%s\n" emacs-version))'
    30.2
    0.07user 0.03system 0:00.13elapsed 78%CPU (0avgtext+0avgdata 46064maxresident)k

    $ /usr/bin/time ~/bin/emacs --batch -eval '(princ (format "%s\n" emacs-version))'
    30.2
    0.02user 0.01system 0:00.04elapsed 95%CPU (0avgtext+0avgdata 57728maxresident)k

GUI Emacs on a 12 year old processor (i5-4590) feels faster than on a M4 Pro Macbook. I think it's just something to do with the window manager on each of the systems (my experience is mostly with Wayland KDE) rather than the speed of the CPU.