I would like to see an independent benchmark.

This looks quite serious (which would be unsurprising given that Fei-Fei Li and Christopher Ré are involved).

I'm also quite interested in the nuts and bolts: does anyone know what the current accepted leaderboard on this is? I was screwing around with GritLM [1] a few months back and I seem to remember the MTEB [2] was kind of the headline thing at that time, but I might be out of date.

[1] https://arxiv.org/pdf/2402.09906 [2] https://huggingface.co/blog/mteb