Remix.run Logo
▲ bob1029 2 hours ago

Distillation comes in many forms.

Hypothetically I can put a black box around their black box and log every single token in and out as part of ordinary business. This can be a profound amount of data. I can turn around and train my own model using this data. How would anyone detect this? Reasoning tokens are only so much of a moat.

▲joshuamorton an hour ago | parent | next [-]

The point of distillation is that it's a much more efficient form of learning.

Indeed the theoretical form of distillation for a teacher & student model is that at each token you can backprop over the entire distribution of outputs instead of the actual chosen output, which makes learning faster.

The point isn't to make copying the model impossible, it's to make it computationally (or financially) infeasible.

▲bertonvv an hour ago | parent [-]

>The point isn't to make copying the model impossible, it's to make it computationally (or financially) infeasible.

The irony is that compute needs to get cheaper over time, for these companies to train the next bigger model and not go bankrupt. Their survival depends on distillation becoming more feasible over time.

▲xgulfie an hour ago | parent | prev [-]

This is a great idea tbh. Just offer the tokens at a discount and you've got yourself a business model.

▲a34729t 36 minutes ago | parent [-]

Yuuuuuuup. A token harness, if you will.