| ▲ | croemer 2 hours ago | |||||||||||||||||||||||||||||||
> I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL We don't know if it's still important or not. Cross-entropy going down could just be a correlate of whatever else matters. The graph doesn't show causation: that perplexity is what makes them useful. | ||||||||||||||||||||||||||||||||
| ▲ | jonahx 2 hours ago | parent [-] | |||||||||||||||||||||||||||||||
The idea that compression ~ intelligence has other support: | ||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||