Did you try providing it documentation for the respective formats (via browsing/tool use or input to the prompt)? And were you using a modern thinking model from Anthropic or OpenAI?

The crucial breakdown here sounds like either lack of proper context/harness or insufficiently capable model (there's a huge gulf between GPT-5.5/Opus 4.8/Fable class models and anything not from the big three) or both.

That's exactly what the people in my orbit and whom I'm watching are doing, and some of their outputs are fueling the excitement.

If you aren't seeing remarkable things being done with this tech, I'd argue you aren't looking hard enough. I understand there's a lot of noise obscuring the signal, but that's always the case with a "big thing."

Something a lot of folks struggling with these systems don't get is that the instruction and management of them is often quite important - just because they're capable doesn't mean they're mind readers.

Most of the skepticism I encounter on this front is due to lack of proper direction, process involving planning and review before execution, and appropriate attention given to evaluation and feedback loops.

If you asked the smartest person in the world to YOLO a task with the sort of instruction the average denier uses to evaluate an LLM, you'd likely find they wouldn't get back what they were expecting either - and if you're evaluating on subpar models/tools, you shouldn't be surprised to get subpar results.

The same attitude has been directed at points through history for people "who depend on the internet," "who depend on computers," and "who depend on machines."

I was told growing up "you won't always have a calculator in your pocket" and yet now my phone has an offline LLM on it.

You’re absolutely right! This would save so much time.

Sent from my Claude Code

Totally agree that the lack of a common base of evaluation is terrible for the discussion, and benchmaxing only contributes to this.

The only way to get a sense for these systems is to use them on things you know well, and everyone knows different things at different levels.

People also tend to underestimate how fast this is moving and base their take on dated and subpar systems for a variety of reasons, a key one being that the firehose is too big for any one person to have a proper focus on all of it.

> AI fans hail it as huge shift in whatever metrics the AI vendor has gamed this time

Hint: Those are not AI fans. (See the top comment for context.)

It shouldn't be a surprise that the baseline for "best" shifts as better tech comes out, but that doesn't make dated models any less capable than they were when they came out.

Skeptics continue to move the goalposts on what constitutes this mattering, but the fact that frontier systems are making novel maths & sciences discoveries and I can run an LLM on my phone for simple tasks that would've been unthinkable a few years ago are testaments to the directionality of the tech.

Seriously I've been on a sabbatical and have no pressure to use AI. I keep trying on my compiler and for coding they flail so hard every time. I love how good they are at helping me diagnose bugs, it's so much better than scouring google like I used to hoping for some result that matches what I'm seeing. Even then, they hallucinate all the time and just say odd things that don't make sense. As soon as you ask them something obscure they just.... merp

They're literally doing novel research. The smartest mathematicians in the world couldn't solve Erdős' planar unit distance problem for 80 years, and OpenAI's models knocked that out a couple months ago.

This stuff is moving fast, and if you aren't evaluating SoTA on at least a quarterly basis, you're going to have a bad time.