Remix.run Logo
▲ joshuamorton an hour ago

The point of distillation is that it's a much more efficient form of learning.

Indeed the theoretical form of distillation for a teacher & student model is that at each token you can backprop over the entire distribution of outputs instead of the actual chosen output, which makes learning faster.

The point isn't to make copying the model impossible, it's to make it computationally (or financially) infeasible.

▲bertonvv an hour ago | parent [-]

>The point isn't to make copying the model impossible, it's to make it computationally (or financially) infeasible.

The irony is that compute needs to get cheaper over time, for these companies to train the next bigger model and not go bankrupt. Their survival depends on distillation becoming more feasible over time.