| ▲ | foohis 23 minutes ago | |||||||
Anthropic spent a lot more time editing their pre-training data with higher-quality material. For example, they used actual books for pre-training, because books are generally run through human editors for higher quality. (this is from the copyright lawsuits) Other companies would use completely raw internet data for pre-training, which is basically trash. Anthropic's secret sauce is that they filtered and edited down their data for only the highest-quality material. You really don't need quadrillions of tokens to pre-train, as that's just random noise. A smaller-but-higher-quality data set is better in every way. Meanwhile its obvious that companies like Google would use trash data like email and chat logs or Internet forums for pre-training. Just complete trash data. This is a classic case of "less is more" and "garbage-in-equals-garbage-out" | ||||||||
| ▲ | quixoticelixer- 10 minutes ago | parent [-] | |||||||
That doesn't really make sense because as good as Opus 5.5 is it writes awful terrible, disgusting slop. | ||||||||
| ||||||||