| ▲ | namr2000 3 hours ago | |||||||
While intuitively it would make sense that smaller models specifically trained on math perform better than large generalist models, this has empirically proven to be untrue. Deepmind was proving math theorems using a model specifically trained on math, and that was very quickly surpassed by generalist LLMs. It's presently not well understood why this is the case. | ||||||||
| ▲ | rednb 2 hours ago | parent | next [-] | |||||||
I'm surprised that this is surprising for the industry. All knowledge proceeds from the same wisdom. The more fields you master at a very advanced level (eg. STEM, philosophy, marketing, etc...). The more you notice that the same basic principles drive seemingly unrelated fields. I am not simply talking about applying a mathematical technique commonly used in one field to another. At a deeper level, you always get back to playing the same game in a different form. For example, sales engineering has a lot in common with cybersecurity. And this is a good example, as it's extremely rare to find someone with advanced expertise in both fields, since they typically attract two radically different kind of people. For this reason, where the output expected is not mechanical, i'm not surprised at all that generalist models are doing better than specialized ones for open ended questions and frontier work. | ||||||||
| ||||||||
| ▲ | kypro 10 minutes ago | parent | prev | next [-] | |||||||
If I were to guess it's because a model trained purely on maths wouldn't be as likely to create such diverse and deep layers of abstractions as it would need to if it were being trained to predict something as complex as the real world. A maths model trained only on maths would only learn how to predict something if a pattern was exposed often enough and was a useful enough predictive signal. While I'm sure it could all the same patterns and abstractions with a maths dataset alone, it would depend far more on that dataset being exceptionally well designed. As an example imagine a self-driving car. A self-driving car needs to understand road signage. You could train a self-driving car with all of the signs in the US and I'd suggest that it wouldn't be as good at predicting signs in the real-world as an AI which in addition to being trained on road signs, could also read text and understands material science. Why? Because in the real world signs can be damaged and distorted. An AI that could understand the ways in which signs might distort and can understand that a four letter word beginning with an S and ending P is probably "STOP" even the T and O are ineligible. Of course, an AI with a rich enough dataset containing damaged signs might eventually get equally as good, but perhaps would never quite understand the patterns that underlie the patterns - it would recognise all of the way a stop sign can be damaged and distort, but not why the distortion happens or what the letters on the sign are and how they relate. | ||||||||
| ▲ | hn_throwaway_99 25 minutes ago | parent | prev | next [-] | |||||||
Correct me if I'm wrong but I thought there were still a number of cases where models trained for specific tasks perform much better than frontier generalized LLMs. E.g. Stockfish and KataGo perform much better at chess and go than the best frontier models (obviously I don't know what the internal models are capable of in these realms). | ||||||||
| ▲ | davidguetta 2 hours ago | parent | prev | next [-] | |||||||
general training remove noise | ||||||||
| ||||||||
| ▲ | trash_cat 2 hours ago | parent | prev [-] | |||||||
Sounds like a bitter lesson. | ||||||||