Remix.run Logo
slowin 4 days ago

I'd love to! For real coding though, SOTA models barely get the job done. It wasn't until Opus 4.5 that you could really get decent results.

I'm sure this will change (and I can't wait for it!) but as of today, open models might be fine for summarizing and writing docs, but you need SOTA to work on code if you want to be competitive.

nemomarx 4 days ago | parent | next [-]

Every time I check in on this I hear a more recent model is the one where they started doing good work. I'm excited to here that Astra is where it got capable enough to work on code next year

tomashubelbauer 4 days ago | parent [-]

I think it truly just was Opus 4.5 where LLMs became usable for coding.

redox99 4 days ago | parent | next [-]

4.5 became useful for one shotting large features.

LLMs were useful for coding ever since GPT3 (copilot), and sonnet 3.5 for agentic coding.

zackify 4 days ago | parent | prev [-]

Yeah what, I'm using flash models and getting a ton of stuff done. 300-400b param size with pi.dev

kbwal7 4 days ago | parent | prev | next [-]

I do think there are now open weight models that are on par with (or beating) Opus 4.5 by now (e.g. Kimi K3, GLM5.3). But yeah obviously the frontier closed source models seem to have pulled away once again, so open weight seems to be a few months behind right now (which might be too long to wait for a lot of people!).

redox99 4 days ago | parent [-]

Those two you mentioned completely demolish opus 4.5. It's not even close. I'd say they are between opus 4.8 and opus 5. And better in some tasks.

happycube 3 days ago | parent | next [-]

And if you run it yourself, you know it will be generally consistent with no random quality dips.

water-drummer 4 days ago | parent | prev [-]

Opus 5 feels like a regression compared to 4.8

horsawlarway 4 days ago | parent | prev | next [-]

You're not corporate America (and trust me, I mostly mean that as a plus).

I also work in software, and while I vaguely disagree that open models can't be used (they absolutely fit into productive niches here, and holy hell are the last generation [ex laguna s1, kimi k3, glm 5.3, etc] actually decent) - I will agree that SOTA are a better fit for software development, especially when used in conjunction with an already very expensive employee who's driving them.

But for "Corporate America"... no. You absolutely don't need SOTA. They're doing things like transcription, summarization, customer interaction, minor technical tasks like form creation in existing tools, report generation (ex - powerpoint, pdf, docs, etc) and other general "white collar tasks". Think about roles in business that are in the 60-85k compensation range.

It's mostly busy work that keeps existing processes flowing and the business on the rails. Important, but not research/novel.

And cheap ai... is a wonderful fit for a lot of this. No one wants to replace an employee making 80k with a less reliable AI that costs 45k a year in tokens (SOTA). But they're absolutely willing to drop 2-3k/year on AI (~100/month - right in the open model cost range) for that employee if they can get a 10% bump in productivity or happiness.

ThrowawayR2 4 days ago | parent | next [-]

Greg Kroah-Hartman from the Linux kernel team said recently in a talk (https://www.youtube.com/watch?v=_MwMLPmMccs) running open models locally is already good enough for finding Linux kernel bugs and generating patches to fix them. He said that the false positive rate he's encountering was only one third to one quarter IIRC. He doesn't provide any details but it didn't sound like he was running some kind of huge server or something either, just a desktop workstation of some sort.

patja 4 days ago | parent | prev | next [-]

"real coding" carries a lot of the weight in that comment.

Seems like if you ask 5 different people what "real coding" means you might get 5 different answers.

Not everyone is building the next framework or compiler.

Self-hosted Qwen 3.8 @Q4 on my RTX 3090 can produce beautiful functional CRUD pages and apps all day long. And that is 90% of the "real coding" being done in corporate settings.

The quote in the article about Mazda vs. Maserati captures this. Many might want the Maserati and drool over its specs and capabilities, but balk at the cost and how often are they really going to run it up to full performance limits on their daily commute to their cubicle?

slowin 4 days ago | parent | prev | next [-]

Yeah, I totally agree. I'm sure these open models are more than good enough for non-dev work. I'm also sure they'll be good enough for dev work soon enough (and some people are saying the latest already are). My point was that given the difficulty of writing great code and dealing with large systems, SOTA just recently emerged as a viable option. I expect open models to catch up soon.

tinyplanets 4 days ago | parent | prev [-]

Good luck to the poor shleps trying to make a living performing "white collar tasks" I guess, right? They can all go be poets or painters...

spopejoy 3 days ago | parent | prev | next [-]

I'm having no difficulty getting deepseek v4 to blast out good code. I guess devs want to outsource all of their thinking now? Yes open models maybe can't design the whole thing soup to nuts but why is that necessary?

transdev12 4 days ago | parent | prev [-]

I generally use sonnet 5 for most coding tasks, a lot of coding is really pretty straightforward.