Astra was insane until Monday but something happened on tuesday, now it feels like Sol. I grieve for the lost productivity but i hope they may give us the original Astra back.

▲ sobellian 5 hours ago | parent | next [-]

I thought the same, but on second thought I merely had to deal on Tuesday with a lot of the mistakes Astra made on the preceding days. I wonder if this time lag of consequences explains why the sentiment is so common with these models. It probably also cautions against irrational exuberance when you first crack open a new model and it one-shots various problems, as you don't yet know what goats Astra had to sacrifice to make it so.

▲ cainxinth 7 hours ago | parent | prev | next [-]

It's the same story every time OpenAI or Anthropic releases a new model. They are generous with compute for the first few days, and use maximum fidelity with uncompressed weights. Everything runs at its best to make a good first impression. But eventually they pare things back and the models perform a little worse.

▲ Vetch 5 hours ago | parent | next [-]

The most charitable explanation I can think of for this is something like regression to the mean. When a model is first released, there'll be a subset of users who, just by chance, sample the highest quality band of the distribution that answers their query. Some of them will rush over to social media and post about how amazing a model is. Over time, those users' mental model of responses will converge but they'll perceive the model's return to typical performance as a downgrade.

This guess/explanation predicts that most users won't match what the initial social media hype claims, doesn't discount user experience as simple habituation nor does it assume companies are lying when they say there have been no changes to the model itself (quantization included).

I also think there's an aspect where initial testing is more forgiving because the more persnickety polish bits can be ignored and tests are likely to have similar structure to things that can be trained for. Meanwhile, actual specific work items are a broader unusual distribution with more stringent acceptance criteria.

Personally, I can detect a separation between Sol and Astra (but not as large as that between Opus and Fable). While they can solve most of the same problems, Astra takes less time, is less frustrating to talk to, is cleaner, notices more, spins wheels less and requires less corrections.

▲

roywiggins an hour ago | parent | next [-]

Another effect may be that with newer models, people try out the hard problems they got stuck with in older models, and when that model happens to succeed, they are very impressed.

Perhaps the new model really is better... but perhaps merely being prompted to try again with the difficult problem just gave them a new dice roll and they came up lucky. And then regression to the mean kicks in.

▲

kmeisthax 3 hours ago | parent | prev [-]

To add onto this, if you use a shiny new model and it gives you a turd, you're not going to tweet about it ("hey guys, look what I made with Astra! Nothing!"), and even if you do nobody is going to interact with it so it does poorly in the algorithm, because it has to compete with all the people using the new model to make something that looks impressive. Then people get tired of the magic trick and the logic flips.

	▲	zaphirplane an hour ago \| parent [-]
		Really? there would be complaints, it’s expensive and doesn’t do as well

▲ baby 6 hours ago | parent | prev | next [-]

You think they introduce stronger quantization after a few days?

▲

boredatoms 6 hours ago | parent [-]

For sure they quickly move to q8, the output quality difference to bf16 is small compared to the speed/capacity gain

▲

NineStarPoint 5 hours ago | parent | next [-]

Yeah q8 made so littler difference back when I was testing such things I'd be surprised if people could quickly notice that as a change. It's got to be either further quantized or some other type of optimization that kicks in when people notice the drop.

▲

selectodude 4 hours ago | parent [-]

NVFP4 would buy them a huge increase in capacity but I think it would be noticeable.

	▲	Caracas288 4 hours ago \| parent [-]
		Why doesn't someone just try to measure this next time!?

▲

torginus 4 hours ago | parent | prev | next [-]

Some people here have remarked previously that while reduced precision doesn't show up in quick prompts, it does severely impact these models' ability to perform long running tasks - to the point that running these big models with severe quantization might be counterproductive as smaller but less quantized ones perform better.

▲

nonethewiser 4 hours ago | parent | prev [-]

Could this explain Opus?

▲ holoduke an hour ago | parent | prev | next [-]

I am sure every input send to openai is prechecked by a dumb model and then send to another one. They heavily tweak this to improve performance.

▲ blurbleblurble 6 hours ago | parent | prev | next [-]

Or a lot worse

▲

holler 6 hours ago | parent [-]

so, AGI is cancelled?

▲

__MatrixMan__ 6 hours ago | parent [-]

AGI for the peasants is cancelled.

	▲	elwell 2 hours ago \| parent [-]
		Trogdor - the AGInator

▲ dooglius 5 hours ago | parent | prev [-]

Do you have hard evidence of this assertion?

▲ simlevesque 5 hours ago | parent | next [-]

We can't have hard evidence. It's a SaaS and they own the code and the machine it runs on.

So it may be a widespread hallucination. But there's no evidence of that either.

▲

dooglius 2 hours ago | parent | next [-]

Run a benchmark with a large number of samples, rerun a few days later. Compare results, use statistics to see if there's a statistically significant difference.

▲

fragmede 4 hours ago | parent | prev [-]

We could still have soft evidence though. Make a Todo app on Monday, and make a Todo app on Tuesday, and see what it makes in comparison.

▲

marcus_cemes 4 hours ago | parent | next [-]

You would need a significant sample size to make any sort of conclusion from such a probabilistic process. Then there's the issue of how you would actually grade/compare.

▲

ArvidSu 4 hours ago | parent | prev | next [-]

You only need to come up with a catchy "SomethingBench" name, post it on reddit/x and now you're an ai sage. Not to disparage the launch/after comparison though, I'd genuinely enjoy a data point like that

▲

luckydata 4 hours ago | parent | prev [-]

someone already does that https://aistupidlevel.info/

	▲	chaimtweiss 3 hours ago \| parent [-]
		It's actually a extremely cool site, and fascinating to view the results off the AI bots i use.

▲ bradly 5 hours ago | parent | prev [-]

There is a toot from an Open AI person a couple days ago saying they are "pulling all the levers" because of capacity issues. I have no idea what the heck the person is talking about, but I'm guessing there are consequence for those levers.

    > "Demand for Astra is really unprecedented. We're pulling all the levers possible to sustain the demand, but I've not seen anything like it until now and we went through very steep growth before. Priority will always be to keep excellent service for existing users, but we might have to pause new Pro subscriptions for a bit if this continues."

	▲	nonethewiser 4 hours ago \| parent [-]
		Depends on the nature of the levers

▲ scrlk 6 hours ago | parent | prev | next [-]

Might be related to this announcement from Tibo on Sunday:

> We've made some improvements that improve usage on the long tail for power users of Astra when logged in with your ChatGPT account.

> No change in quality and a pure win that on the long tail can result in up to 3-4X less usage being drawn from the subscription.

https://x.com/thsottiaux/status/2096717905614524491 (https://xcancel.com/thsottiaux/status/2096717905614524491)

▲

siva7 6 hours ago | parent [-]

It seems to me the people working at OAI may believe all other humans must be a little bit behind intellectually.

	▲	rowanG077 an hour ago \| parent \| next [-]
		It always reminds of the story of the creator of counter strike. Every new release he would get a ton of complaints from players about things they didn't even change. Notably that each version had more lag. And he got so fed that he start to negatively subtract peoples pings. And suddenly a ton of players reported back that the change was incredibly good. Point is, I really don't buy all the stories about a model suddenly being downgraded without at least a modicum of substance. People are grasping at straws in the noise.
	▲	pixl97 5 hours ago \| parent \| prev [-]
		I mean, in general they aren't wrong. You can't fool everybody all of the time, but you can fool almost everybody most of the time. But most of all, it's easy to fool yourself.

▲ binary0010 6 hours ago | parent | prev | next [-]

Disagree completely. I started using Astra from Sol the day it was released, and was a virtually imperceptable difference and made lots of mistakes and shit architecture decisions from day 1 of release.

▲ theLiminator 6 hours ago | parent | prev | next [-]

I wish someone ran some sort of representative benchmark suite every X days to see if this occurs.

▲

micycle1 6 hours ago | parent | next [-]

https://marginlab.ai/trackers/codex/

	▲	CamperBob2 6 hours ago \| parent [-]
		Unfortunately that seems to be monitoring Sol, not Astra, unless I'm missing something.

▲

embedding-shape 6 hours ago | parent | prev [-]

I'm not sure how you could run such a benchmark without leaving it possible for the labs to easily detect and fudge the results.

▲ mccoyb 6 hours ago | parent | prev | next [-]

I had nearly the exact same experience and thought I was imagining it … absolutely ripping, then it turned into Sol++ on Tuesday …

I’m working on hard things, it is very noticeable when it is hums through something and then falls over on something it should not

I can tell by analyzing my own prompts to look at when I get frustrated ;)

▲ throwatdem12311 5 hours ago | parent | prev | next [-]

I’m so used to seeing this on every single model release I’m starting to question if these kinds of posts are just trolling.

Alternative theory - it always seems amazing when it first comes out then the novelty wears off and we’re just meh about it. New model is a model is a model. I bought a PS5 Pro and was genuinely blown away by it at first…few weeks later I’m just like…eh it looks pretty good I guess? It’s still the same, I’m just used to it now and the wow factor along a new thing is going. Kinda like that.

Or they are just compute constrained so they have to serve a shittier version. Who knows?

I hate how opaque these companies are. It feels deceptive and evil.

▲ konart 4 hours ago | parent | prev | next [-]

>now it feels like Sol

It can very well be Sol, no? What stops them from using cheaper model for some requests during "rush" hours or simply use cheaper model for every Nth request.

▲ nickreese 7 hours ago | parent | prev | next [-]

I had the same experience. Moving back to Sol for actual implementation.

▲ jcmontx 7 hours ago | parent | prev | next [-]

Same story every time, I bet they quantized it

	▲	manmal 7 hours ago \| parent [-]
		Exactly my thoughts today. They have to make it cheaper after demoing what’s possible initially.

▲ qaq 4 hours ago | parent | prev | next [-]

OK so it's not just me

▲ acedTrex 4 hours ago | parent | prev | next [-]

This shit is just vibe coder astrology lol

▲ Razengan 6 hours ago | parent | prev | next [-]

Which plan/region are you on/in?

	▲	siva7 6 hours ago \| parent [-]
		Highest subscription tier and i believe there is only US region available being served globally

▲ ModernMech 6 hours ago | parent | prev [-]

lol I didn't get access until Monday (I was at 0% since Friday and my reset was Sunday at 11pm), so go figure.