Distilling from a closed model like GPT-4 via API would be architecturally crippled.

You’re restricted to output logits only, with no access to attention patterns, intermediate activations, or layer-wise representations which are needed for proper knowledge transfer.

Without alignment of Q/K/V matrices or hidden state spaces the student model cannot learn the teacher model's reasoning inductive biases - only its surface behavior which will likely amplify hallucinations.

In contrast, open-weight teachers enable multi-level distillation: KL on logits + MSE on hidden states + attention matching.

Does that answer your question?