I'd be interested to know what was going on with it during the public test as there were numerous reports of it improving considerably at tasks it was asked to do early on in the test compared to later in it.

▲

utilize1808 3 hours ago | parent | next [-]

It's logical to serve the best version (quant) of the model at the beginning so that users keep testing it. It is also reasonable to think that the developer of the model tried to test various quant levels by gradually degrading the model's capabilities.

	▲	brookst 2 hours ago \| parent [-]
		I mean that’s imaginative but not sure there’s any evidence at all for it, and it’s the opposite of what the comment you replied to observed.

▲

daveyoung 3 hours ago | parent | prev | next [-]

Two potentials from my pov:

1. Just variance in pass@K. If you prompt any model multiple times you'll see a large variance. N=1, but I find chinese open source models have a higher variance than higher-RL'd models like fable/opus.

2. They legitimately shipped a new RL checkpoint over the 7 days, which I find hard to believe.

I am leaning towards 1.

▲

dannyw an hour ago | parent | next [-]

Or 3, they find some bug/regression in their pipeline; maybe they didn't quant parts of a model properly, maybe their inference engine had a bug, maybe some pinned MoE expert wasn't pinned, etc...

That's very plausible to have, identify, and fix in a day; especially when you get community feedback in the wild.

▲

zarzavat 3 hours ago | parent | prev | next [-]

3. Deployment problems unrelated to the weights causing degraded performance

	▲	swiftcoder an hour ago \| parent [-]
		For sure the version accessible from OpenCode had a massive timeout problem the first day or so, which seemed to heavily degrade its task completion rate

▲

re-thc 3 hours ago | parent | prev [-]

2. There was a new checkpoint. Official.

	▲	daveyoung 3 hours ago \| parent [-]
		do you have reference to where it was said?

▲

rfoo 3 hours ago | parent | prev [-]

lol don't shout out the obvious