I would love to see a stackexchange-like site where humans ask questions and we get to vote on the reply by various LLMs.

is this like what you're thinking of? https://lmarena.ai