> It is widely known that an LLM can only find data it has seen before.

Maybe an editing snafu? It is odd.

I think in context they meant something like 'it is unsurprising that an LLM can find data it has already seen before.' The intent is to point out that benchmarks tend to use public data. In context I don't think they were actually trying to claim LLMs are useless at unfamiliar data, just that the benchmarks have a specific flaw.