Remix.run Logo
▲ iagiagiag 24 minutes ago

> We do not expect it is possible to achieve a loss of 0, which would correspond to being able to predict the dataset entirely deterministically.

Why not? Models are already overfitting a lot, I don't see the problem if they were made to totally overfit on the whole dataset during that 1st training phase. I don't know if that would be useful (or easy to implement) but I don't see the issue. Only in the "finetuning" phase you would decide if an information was useful or not to retain.

Step (1) you compress human knowledege, with or without loss, Step (2) you train the algorithm to use it, and you might discard what you don't want in it.