Remix.run Logo
▲ foohis 23 minutes ago

Anthropic spent a lot more time editing their pre-training data with higher-quality material. For example, they used actual books for pre-training, because books are generally run through human editors for higher quality. (this is from the copyright lawsuits)

Other companies would use completely raw internet data for pre-training, which is basically trash. Anthropic's secret sauce is that they filtered and edited down their data for only the highest-quality material. You really don't need quadrillions of tokens to pre-train, as that's just random noise. A smaller-but-higher-quality data set is better in every way.

Meanwhile its obvious that companies like Google would use trash data like email and chat logs or Internet forums for pre-training. Just complete trash data.

This is a classic case of "less is more" and "garbage-in-equals-garbage-out"

▲quixoticelixer- 10 minutes ago | parent [-]

That doesn't really make sense because as good as Opus 5.5 is it writes awful terrible, disgusting slop.

▲foohis a few seconds ago | parent [-]

It's a much, much better writer than literally every other model.