What is doing the intelligencing though? Is it the LLM or the person training it?

To me, that seems awfully close to arguing that a puppet is intelligent because a human is pulling the strings and making it dance.

We can agree to disagree on this.