This isn’t true. You can look at basically every cross encoder used today and observe degradations in precision with increases in k

Ofc latency matters for retrieval pipelines and this is another reason to care. But first pass retrieval has to surface the right candidates for it to matter at all. It has to do it within the constraints of the precision degradation wrt k of the first pass reranker