If you go to a LLM without harness, GP original point in completely right.

LLMS by themselves are still shit at math, they still confuse weird correlation to causation every time (and sometimes in ways even a 9 year old would say "no, that's dumb"), and confuse original parameters very often.

I disagree with " "thinking" token generation being on the correct track just to 'no, wait' on an already correct conclusion", because i think that is an effect of the harness, not the LLMs.

80% off all the improvements since ChatGPT4 are in the harnesses, and the LLMs by themselves, while they improved in areas they already were good at (translation especially) did not fix any of they original issues (object permanence, calculusm correlation).

Just run old models in the playground and get them to play chess (maybe make a small custom harness if you feel like it), then replace it with a frontier model (i don't know if you still have API access without harness on US models, but if you don't try K3), you will see LLMs weaknesses were not at all fixed, even marginally. They're way better and not inducing bugs in the code, so that make them usable since Opus4.5 (anyone using them prior to that either had a greenfield project or like spending hours debugging).

I agree, harnesses is where everyone improved the most. Our internal experimental tool can now semi-reliably formulate small programs to assert the correctness of their theories, for example. Context length is still a weird factor that we haven't sensibly solved - if anything, the lesson learned was to keep the context as small as possible and do most of the true "thinking" in the harness and temporary generated code.