Shouldn't matter when I am not on GUI seat. In my SSH session with X11 forwarding there is no DISPLAY emacs could use.

Tried it anyways, looks the same:

  $ time emacs -nw -Q -e kill-emacs
  real    0m0.075s
  user    0m0.062s
  sys     0m0.013s

s/with/without/