Remix.run Logo
bbg2401 5 hours ago

This is going to sound accusatory but I promise it isn’t.

Did you feel the same way about 5.5? The same general sentiment keeps being expressed with every model release. The prior generation is immediately cast aside with a vague “well yes, actually we didn’t mean it last time” attitude.

I think I even heard Theo the T3 guy suggesting that his work comes to halt if he can’t use the latest generation model, despite heaping significant praise on each generation previous and suggesting it does all of his programming tasks.

The boundless energy with which each model generation is said to be revolutionary and leaving everyone behind is tiresome even when incremental progress is real and measurable.

jimmaswell 3 hours ago | parent [-]

Prior to 5.5 I thought the models were very useful but not trustworthy for certain tiers of tasks and needed more in depth verification. Frequent hand edits and times where it felt more productive to just finish something myself than keep rolling the dice on the model not subtly messing it up. But still great for targeted things.

5.5 on Extra High I found to be a huge leap in what it could tackle and how effectively but only up to a point. Still had common failure modes here and there but a big improvement

When I tried 5.6 I wasn't expecting more than an incremental upgrade, but it blew me away with how "smart" it seemed, how it completely sidestepped failure modes I'd been used to dealing with in 5.5, its success rate on the first try for wide-sweeping architectural tasks, and how I've basically never seen a bad hallucination make it through to the final output. It's the first time I've felt like if I left it with a task on the highest setting and it had any way to verify its output, and told it to keep going, it could solve almost anything. This could all be incremental upgrades in many different areas of the model+harness, but the cumulative effect feels like crossing that "escape velocity" threshold. The point where there are no more rough edges and it "just works".

I can't speak to others, but my feelings on all of these have been consistent since I tried them in the first place. I've also heard model providers draw people in at each new release with the model at full performance and then degrade it over time hoping users won't notice, which has seemed plausible anecdotally but I have no hard proof.