Hi Antoine!

Interesting point about backend variance. Do you think serving layer should become part of standard LLM eval reporting?

Hi! Yes, I definitely think so. I've seen variance across all model families I looked at. The magnitude changes, but the presence of variance is a constant.