| ▲ | andai 5 hours ago | |
I remember reading something a few years ago, about how if you train an LLM with the reading material sorted by grade, the training becomes more efficient? Does anyone know about this technique? How does that work? I'm assuming the knowledge doesn't end up as separate "layers". I'm also reminded of how the human mind develops in distinct stages (e.g. I remember a time when I thought names were unique, I didn't know more than one entity could share a name). | ||
| ▲ | NitpickLawyer 4 hours ago | parent [-] | |
> Does anyone know about this technique? How does that work? The term you are looking for is Curriculum Learning. There are several papers exploring this. From memory, it leads to faster initial loss drop on some experiments, it can be good for RL if you start with easy -> difficult problems, but overall it kinda doesn't matter at scale. (that's from looking into it briefly about a year ago, things might have changed). | ||