The argument that models are conscious only during training, and not conscious during inference feels like a painful one to make. Are we killing models when they reach the training objective?

Technically we are killing models when they don't reach the training objective by throwing those weights away via survival of the fittest (fit meaning what we humans think we want).

Kinda like creating quantum copies of your child and keeping the ones that answer correctly and shooting the other ones in the face.