Remix.run Logo
jimmaswell 3 hours ago

Prior to 5.5 I thought the models were very useful but not trustworthy for certain tiers of tasks and needed more in depth verification. Frequent hand edits and times where it felt more productive to just finish something myself than keep rolling the dice on the model not subtly messing it up. But still great for targeted things.

5.5 on Extra High I found to be a huge leap in what it could tackle and how effectively but only up to a point. Still had common failure modes here and there but a big improvement

When I tried 5.6 I wasn't expecting more than an incremental upgrade, but it blew me away with how "smart" it seemed, how it completely sidestepped failure modes I'd been used to dealing with in 5.5, its success rate on the first try for wide-sweeping architectural tasks, and how I've basically never seen a bad hallucination make it through to the final output. It's the first time I've felt like if I left it with a task on the highest setting and it had any way to verify its output, and told it to keep going, it could solve almost anything. This could all be incremental upgrades in many different areas of the model+harness, but the cumulative effect feels like crossing that "escape velocity" threshold. The point where there are no more rough edges and it "just works".

I can't speak to others, but my feelings on all of these have been consistent since I tried them in the first place. I've also heard model providers draw people in at each new release with the model at full performance and then degrade it over time hoping users won't notice, which has seemed plausible anecdotally but I have no hard proof.