We could call this "generative adversarial network" (GAN) :)

https://en.wikipedia.org/wiki/Generative_adversarial_network

Why supply underspecified requirements in the first place? Both models are good at challenging assumptions/edge cases and asking questions to clarify, but seemingly only when explicitly asked (i.e. something like a "brainstorm" skill).

I don't think either harnesses do enough to encourage the model to challenge all assumptions and ask questions, maybe because users might find it annoying. That step is basically a requirement IMO.

I've found all of the GPT-5 models to be very nit-picky, useful for code review and mathematical analysis, but seemingly gets in the way of "aesthetic" code, e.g. overly defensive code to cover all edge cases, even if unlikely.

There is seemingly also a tradeoff between flexibility vs instruction following. In my experience Opus will sometimes ignore instructions but can "fill in the blanks" more, vs GPT-5.5 follows instructions better but perhaps at the cost of rigidity.

same observation here opus 4.8 (and i dont understand the people defending gpt 5.5 constantly) was significantly mature, it would even push back against anything off putting where as GPT 5.5 will happily agree and do what is asked but I would note that it takes several tries.

4.8 also requires more than one prompt but its output is significantly higher quality and offers more insight

Fable 5 is a different beast however.

> It's so good at taking underspecified requirements and filling in the gaps with sensible approaches for your project.

At a high level. It misses low level or other non-functional requirements differently so I wouldn't say Opus is just strictly better.

It's also possible that it's just a harness problem more than model.

> who are these people and what makes them more qualified than any of you

Anyone can run something and make a web page. These people just do it instead of questioning. Main difference. If everyone asks "how could you" "are you qualified" then we have nothing but gatekeeping.

presumably whatever the top model uses and then some, since the human can use the model.

I wonder if a model could score higher if it had a human at its disposal?

More importantly, I suspect this actually hinders the work. If the LLM does make a mistake, it's now incentivized to downplay it instead of acknowledging and correcting.

The “make no mistakes” admonition does seem pretty silly (it’s been skewered to death on yt), but… it’s easy to imagine how it might work. E.g. it could be interpreted as simply as “check your work”.

Of course, no-one seems to be (publicly) doing the comparative measurements that might allow us to reach rational conclusions here.

Just wait for the next 100 rounds. People love seeing the 65% -> 85% seemingly over and over again for every new model.

Principal-SWE-Bench will take some time to run, because the LLM needs to wait for a crisis to present its solution, having correctly identified that the same solution would have been organizationally impossible to propose until that moment.

As someone who's trying to get better assessments, I'm struggling to come up with objective coding tasks that evaluates all aspects of real life like planning, design choices, problem solving and context usage. From your experience with humans, Do you have any recommendations on what could be effective in measuring it?

I actually really like subjective benchmarks, so long as it's a human (ideally me) grading the results. LLM as judge never made much sense.

> What you really need is an objective benchmark

"When are all the software engineers unemployed?"

Taste is just quality by instinct. At sufficient (and not all that long) timescales, a tasteless product will be more and more difficult to make work at all.

I may be paid to build a machine, but I am a human and take pleasure in arbitrary acts of vanity. I value elegance, and will always favour elegant solutions in engineering and the design of machines, virtual or physical.

That’s the reason why I buy Apple products in private, because I value the design over the exorbitant prices they charge; and it’s the reason why I mull over code that’s already functional until it’s pleasing my ideas of elegance.

I can come up with all kinds of justifications and explanations why the code I’ve written a certain way is objectively better too - understandability matters to the next guy after all - but I won’t be ashamed for taking a certain pride in my work, even if nobody other than me ever values it. That’s fine.

When the LLMs finally take over coding altogether, you’ll have your raw, functional code. Won’t be long anymore. But for now, I’m a human, and I will do human things.

As time passes we will have fewer and fewer literati

Most engineers are wrong (I obviously am the true arbiter of taste), but that doesn't mean there isn't better and worse code.

"Does it work" glosses over a bunch of things: is it fast, cheap, secure, reliable, easy to understand, easy to modify? And that's just for server software where you've nailed down all the functional requirements. Determining what the functional requirements is it's own question.

And all these other non-happy path requirements are somewhat in tension with each other, so what is ideal in one environment is not necessarily ideal in another.

And in particular, "easy to understand/modify" is truly subjective. Different people have different ideas of what easy to understand means. Even if we get to a world where AI is writing all our code, "easy to understand/modify for the AI" is still an important question. We've probably all seen prototypes that collapse under their own weight of slop by now.