|
| ▲ | rytill 3 hours ago | parent | next [-] |
| LLMs learn a distribution during pre-training, not only an average. Then, by giving them context or by post-training, you can make them sample non-average parts of the distribution they learned. |
| |
| ▲ | OtomotO 3 hours ago | parent [-] | | > Then, by giving them context or by post-training, you can make them sample non-average parts of the distribution they learned. How do you derive that something is "below average" or "average" or "above average"? | | |
| ▲ | rytill 2 hours ago | parent | next [-] | | Well, it’s up to the user or post-trainer of the LLM what they believe to be above average. Then they can design around that. In the case of real world LLMs and post-training, what is above average is defined roughly as: labeled good by expert humans, and scoring high on RL environments related to coding like debugging, passing tests, or running efficiently and verifiably correctly. | |
| ▲ | nextaccountic 2 hours ago | parent | prev [-] | | > How do you derive that something is "below average" or "average" or "above average"? One technique is RLHF: have an human expert assess it. |
|
|
|
| ▲ | bozdemir an hour ago | parent | prev | next [-] |
| I dont think Opus 4.8 is an average coder, with my own experience (I have coded 20 + years before even llms existed) it is anything but average. I don't think training data alone determines the success of these models, there are lots of reinforncement learning principles and fine tuning takes place, a crappy code in the dataset doesnt hold those llms scoring high in benchmarks, I dont think an average programmer can score 70% (opus 4.8) in SWE Bench Pro, which is a good one. |
|
| ▲ | piker 3 hours ago | parent | prev | next [-] |
| Which you will necessarily have if they’ve completed a Rust rewrite. |
|
| ▲ | bigupthewhole 3 hours ago | parent | prev [-] |
| You haven't been using AI extensively I presume... I've been programming a long time and considered myself among the top in my domain and AI agents using like GPT 5.5 etc. are much better than me. |
| |
| ▲ | OtomotO 3 hours ago | parent | next [-] | | > You haven't been using AI extensively I presume... Ex falso quodlibet > I've been programming a long time and considered myself among the top in my domain I am not trying to attack you, but you considered yourself that... I don't know whether you actually were and frankly I don't care. | |
| ▲ | witx 3 hours ago | parent | prev [-] | | [flagged] |
|