Remix.run Logo
▲ Xcelerate 2 hours ago

> I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL

From an information theory perspective, cross-entropy loss (log loss) seems to me to be about as pure as it gets in terms of capturing the notion of whether prediction "succeeded". Even Solomonoff Induction minimizes log loss. Would be curious to hear others' perspectives though on why another function might be better for universal prediction purposes.

▲pishpash 2 hours ago | parent [-]

There are more general information measures, Rényi entropy, von Neumann entropy, etc. for different system setups, but for token-by-token information learning and prediction within classical ergodic processes Shannon entropy is where it's at.