I've tried kitty, alacritty, ghostty, wezterm, and konsole (all on X11), but I keep coming back to xfce4-terminal because it opens near instantly and has the least input latency of any of the other terminal emulators I've tried. It also uses very little memory in comparison (with 6 windows open and a fair bit in each scrollback buffer):

  Private  +   Shared  =  RAM used Program
  31.2 MiB +  10.7 MiB =  42.0 MiB xfce4-terminal

I was curious to see what the actual latency was, so I used Pavel Fatin's https://pavelfatin.com/typometer/.

Notes: This system is running Arch, with kernel 7.0.3, scx_bpfland, and i3 on X11. Primary GPU is an integrated Intel GPU.

Using picom:

  Terminal                     Min    Max    Avg    SD
  alacritty                    20.1   111.4  43.8   18.0
  xfce4-terminal               19.1   112.5  49.1   17.9
  xterm                        12.3   152.6  53.1   26.4
  wezterm                      28.3   106.0  56.2   15.4
  kitty                        25.3   115.9  57.2   18.5
  ghostty                      19.2   611.8  63.4   45.5
  konsole                      37.9   138.6  68.8   18.5

  Terminal                     Min    Max    Avg    SD
  xterm                        5.7    63.6   15.2   8.8
  alacritty                    7.4    48.8   20.2   9.7
  xfce4-terminal               8.7    65.3   28.3   12.5
  kitty                        13.2   64.8   30.8   10.3
  konsole                      17.3   71.2   31.1   9.5
  ghostty                      12.9   63.7   33.2   11.6
  wezterm                      18.0   58.6   33.6   11.1

  Terminal                     Min    Max    Avg    SD
  xterm                        4.5    20.4   8.6    1.0
  alacritty                    9.2    13.3   11.1   0.7
  xfce4-terminal               7.4    17.5   11.3   0.9
  ghostty                      11.2   35.3   18.8   3.8
  wezterm                      14.1   34.2   22.7   3.7
  kitty                        12.2   44.3   24.4   5.8
  konsole                      17.2   30.7   25.0   3.6

  wezterm (no picom, NVIDIA)   21.7   51.2   34.8   8.1
  wezterm (picom rt, NVIDIA)   22.0   70.9   41.4   11.2
  wezterm (NVIDIA)             34.4   146.7  69.0   17.8