My gut feel is Anthropic is very technical and pedantic which makes their models really technical and pedantic. They're top at code and technical benchmarks but anecdotally I've found OpenAI to be significantly farther ahead for general usage.

Opus 4.8 will burn 10k tokens trying to answer something 100% whereas GPT-5.5 will burn 2k getting it 90% which is good enough for many things.

Some personal testing on a "help me find that restaurant" prompt https://gist.github.com/nijave/2873b8b10d8c732e46264237b0755...

▲

enraged_camel 3 hours ago | parent [-]

The problem is that the remaining 10% can bite you in bad ways.

I was in Cotswolds, UK a couple of months ago. For those of you who don't know, it's a rural region known for its "chocolate-box" villages and honey-colored limestone architecture. Basically, you go from village to village, most commonly via bus, taking in the sights and doing touristy stuff.

When planning the trip, my sister used ChatGPT, which helpfully (and relatively quickly) found the bus schedules and times for each hop.

Midway through the day, though, we ran into a huge problem: it turns out bus schedules are different on Sundays, and more limited. Which meant we couldn't actually go to our primary destination (the Model Village), and had to cut the trip short.

Yes, ChatGPT was quick and pleasant to use, but missed a crucial detail.

Afterwards I tried it with Opus and it did not make the same mistake.

▲

nijave 3 hours ago | parent | next [-]

Arguably I'd call that the 90%. In my case, answering the restaurant question correctly with "Rishi" in my tests was the sole intent and 90% of the problem. All the models "helpfully" added extra junk about the closure, dates, quotes, etc and many of them got these details wrong--the 10% or extra crap not central to the question.

If the central question was "what is the bus schedule on `day`" and the model screws that up, it gets a fail in my book.

Also curious if Google Maps gets the timetables correct (assuming it has them).

Semi-related, I also discovered that the default web search/fetch tools are pretty primitive and Exa MCP annihilates them. I ended up doing some comparisons with Claude Code comparing built-in server-side to Exa and to a Python MCP that used SearXNG for search and Exa was a clear winner and Python+SearXNG ended up coming out roughly the same after a few cycles of letting Claude optimize the Python code and adjust SearXNG settings. Ultimately it landed on this (making some changes to optimize returning relevant context directly in the search results so the model didn't need an additional web fetch call) https://gist.github.com/nijave/604c43e3e0fdcd60f5280d3a6b109...

▲

deno 3 hours ago | parent | prev [-]

This likely comes down to how it accessed the bus schedules (i.e. web search tool) and not intelligence.

You need to add the actual bus schedule to context somehow (research agent, custom tool or just dump in prompt) and even the simpler modern models will be able to do the planning.

▲

solenoid0937 2 hours ago | parent [-]

Tool usage competency is part of overall intelligence. If the model can't get the information it needs, it must clarify that in the response.

	▲	deno an hour ago \| parent [-]
		This isn't tool usage competency, it's tool quality and/or luck. Regular web search is not good for grounding if you want accurate results. You can ask the model to make a tool for getting bus schedules and then use it only then you are comparing apples to apples in this case.