do you somehow control how non-trivial the queries are? The LLM generates them, right?

what if every engine returns garbage, or on the other hand, handles them too well?

building a benchmark like this in a genuinely fair way seems extremely hard to me. I’m very curious about the details, of course within what you can share.

It is hard to be fair I agree. We tried to be open about what we do here: github.com/keenableai/needle

The queries from what I can tell are not trivial. The actual github repo of the benchmark has a judgement/query browser where you can inspect the different query streams: https://keenableai.github.io/needle/