The people saying that local models can easily replace the real Frontier Models from Anthropic or OpenAI are just not using it for the same things you and I do.

They seem to be happy with their small usages.

For everything I do, i see huge differences in Quality and Speed.

Even when I try an open model on a 4090, the PC gets warm/hot very fast, I struggle with context length.

▲

KellyCriterion a day ago | parent | next [-]

This:

Having currently a very tech-intensive chat running for more than one week, this is where the frontier models really shine: Its no more 2024 or so, where they lost context on large discussions, you can do really great work with the frontier models - the local models will be a thing in a few years, if RAM & CPU developed further, this will happen: in 2035 like, you will be able to run "Claude-like experience" on your iPhone.

▲

NoDodgeQuestion a day ago | parent | prev | next [-]

can you give example, prompt require context window > local model, or higher Quality/Speed, that local model can not do, open source example?

▲

origini a day ago | parent | next [-]

Any program I'm vibing.

I write 5-20 paragraphs of things I want and it goes away and just builds it.

Something like this (just an example):

"Build an application with a webserver in rust. It has 4 stages, first one is an input module... second one is refinement ... third one is an image detection with current fitting ml model ... last one is finetuning."

Then i would describe every basic page with a few more paragraphs.

This would work for half an hour or an hour, consume 100-400k of context tokens and then I would test it, write down 10-50 small sentences (every sentence either fixing a bug, or making it better) which tehn takes again like 20 minutes and i'm nearly done with a complex enough tool providing real value and actually working.

A big kimi, which is hard to run, might be getting close to this but we do not talk about running it on 1-2 5090.

The other thing is, thats more like a practical experience, even if you use them through Open Router (for testing and co), the initial results of something like Opus 5.5 are ALWAYS better. More details, better quality, better understanding of usability of the tool.

▲

the_mar a day ago | parent [-]

I might be misunderstanding but I do the same thing with sonnet, and it runs faster and does well enough

	▲	origini a day ago \| parent [-]
		The discussion was about local models vs. frontier. I would argue that Sonnet is frontier? And yes Sonnet 5.5 is crazy good

▲

simonw a day ago | parent | prev [-]

Here's a prompt I used recently:

> In a work tree build a feature where each column cog action menu has a "Column statistics" item - clicking that item opens a modal dialog which displays statistics about that column - for text columns that's most common values, stats about value length - for numeric columns it's more interesting than that. Let's do a bit of percentile analysis in there too. Under the hood it works by firing off SQL queries using SQLite window functions and suchlike. For the moment let's have that option only available if SQL queries are allowed against the database, and the SQL queries are built in JavaScript.

You can see a screenshot of what Opus 5.5 built here: https://github.com/simonw/datasette/pull/3038

▲

dofm a day ago | parent | prev [-]

> The people saying that local models can easily replace the real Frontier Models from Anthropic or OpenAI are just not using it for the same things you and I do.

> They seem to be happy with their small usages.

Oh yes, we're terribly inferior people. Just living on scraps from the masters' tables.

▲

origini a day ago | parent | next [-]

It wasn't meant as a dismissive comment.

I do plenty of experiments of things i always wanted to do and they just don't work with local models.

Finetuning of some ML models for a specific usecase, some 4k demo idea i had, small browser games to show others game mechanic ideas i had for ages, analysis tools etc.

Even just playing around with some Ghidra eats context like nothing.

▲

yh11 a day ago | parent | prev [-]

[flagged]

▲

dennysora-main a day ago | parent | next [-]

I really strongly agree with this statement. A bunch of people aren't even engineers, and their abilities aren't that good either, yet they keep saying that open-source models can replace OpenAI or Anthropic's models.

If that were really the case, basically nobody would spend so much money using them.

▲

dofm a day ago | parent [-]

Putting aside that you're "really strongly" agreeing with a personally directed insult:

You should factor in that undeniably most people don't have the means to run even a 27B dense model locally, and therefore do not actually know what they are and are not capable of in that context.

And you might want to look at openrouter's rankings.

If you are comfortable automatically assuming that "doesn't use Claude or Codex" is equivalent to "not a good engineer" or "bad engineer", then I think you are probably lost to more nuanced discussions.

▲

dennysora-main a day ago | parent | next [-]

With my DGX Spark + 5080 + 3060 setup, I've tried all kinds of approaches, including quantization, acceleration, fine-tuning, post-training, and everything else.

It still can't compare to Opus 5.5 max in terms of speed, capability, and stability.

And when I calculate all the money I've invested, directly subscribing to the service would actually be cheaper, more efficient, and more stable.

Open source is great, but don't glorify it.

The architecture of the code I develop has never been simple. If you're someone who doesn't even work on things that need to be deployed to something like k8s, then basically, open-source models are indeed enough for you.

	▲	dofm a day ago \| parent [-]
		You can sell your DGX Spark for basically no loss. Maybe even make money.

▲

origini a day ago | parent | prev [-]

I try local models regularly on 2x 4090 and struggle with context length constantly. 48GB on GPU and 128GB from RAM.

the original comment mentioned only 16GB of VRAM.

You know talk about Open Router and 27B dense model?

We are not comparing an even more expensive Hardwaresetup to run local models with proper context window and parameters.

Can i run some open model on a super expensive Mac Book or co with some kind of okayisch results? Yeah sure but you know what the t/s will be and yeah Frontier is still Frontier and Kimi, GLM, DeepSeek are not Frontier. They lag half a year.

You are not getting Opus 5 or Sonnet at home for now.

Why are you so offended by this?

	▲	dofm a day ago \| parent [-]
		I'm not offended. I'm also not the person resorting to insults.

▲

dofm a day ago | parent | prev [-]

Ahh yes. Those are the only two options.