Remix.run Logo
▲ rednb 2 hours ago

I'm surprised that this is surprising for the industry. All knowledge proceeds from the same wisdom. The more fields you master at a very advanced level (eg. STEM, philosophy, marketing, etc...). The more you notice that the same basic principles drive seemingly unrelated fields.

I am not simply talking about applying a mathematical technique commonly used in one field to another. At a deeper level, you always get back to playing the same game in a different form. For example, sales engineering has a lot in common with cybersecurity.

And this is a good example, as it's extremely rare to find someone with advanced expertise in both fields, since they typically attract two radically different kind of people.

For this reason, where the output expected is not mechanical, i'm not surprised at all that generalist models are doing better than specialized ones for open ended questions and frontier work.

▲theptip 2 hours ago | parent [-]

This is definitely a conjecture; it gets to how much SGD will generalize out of the training distribution.

If pretraining alone was sufficient then nobody would do RL. And, heavy RLVR on coding/math is why these models are savant-level in those domains, yet not superhuman at all fuzzy reasoning and practical wisdom tasks that are only represented in the pretraining corpus, and don’t have an RL target.