I think this is actually the opposite in practice. LLMs generate better tests than they are at assessing which tests are effective (and, inferentially, how a system should behave). See: https://arxiv.org/abs/2410.21136

That said, there's also research pointing to LLMs encoding existing bugs as test requirements: https://arxiv.org/abs/2607.05139

That CS PHD researcher, Konstantinou, in particular, has written a lot on this: https://arxiv.org/search/cs?searchtype=author&query=Konstant...