Remix.run Logo
▲ TGower 3 hours ago

> Our new paper (Baron et al., 2026) finds that when the dataset is held fixed, the compute-optimal model size keeps growing with the compute budget.

The dataset is not held fixed in practice. It would be interesting to see an ablation study of where best to allocate compute between more synthetic data, more RL rollouts, and this paper's suggestion of cranking up the parameter count. Unfortunately that's an experiment only a frontier lap could accurately do, and they are unlikely to provide such expensive and valuable information to their competitors.