I mean the act of reproducing the copyrighted material is what is illegal. LLMs I've used for coding has outputted exact copyrights for code verbatim into my code before. When that happens it feels kind of fishy to be honest.

Yes. I agree. But many people argue that training itself is a copyright violation. That's the position I'm countering here.