Remix.run Logo
AmazingTurtle 4 hours ago

gpt-6-astra is a bitch, it constantly scope creeps itself with "yet another thing" to give it that darn polished lick. the results are eventually a little bit better but at what cost? let's do the math.

gpt-5.6-sol: 1x base gpt-6-astra 2.5x base in subscription

then gpt-6-astra tends to spawn subagents a lot, often with all kinds of models such as gpt-5.6, 5.3-codex etc., which is neat. it's a good coordinator but even more cost.

and then it tends to run _full test suites_ over an over again (each costs like 15 minutes) just to verify that _one test_ was fixed etc., and does so for as long as until the test is fixed, eventually accumulating 2 hours or so.

yesterday I assigned it a task to rebase my changs in a repo onto the latest upstream changes. while gpt-5.6-sol consistently took like an hour to do so end-to-end, astra ran for more than 6 hours and still wasn't done. it kept finding "one more thing" that was goldplating that I didn't ask for.

bob1029 3 hours ago | parent | next [-]

> each costs like 15 minutes

I've got a custom agent loop that will reuse unit testing results if no apply patch operations occurred since the last invoke.

Wall clock time isn't something I would put on the AI provider. That's entirely a consequence of the system that you've brought to the party.

jaggederest 3 hours ago | parent [-]

Even better, use some kind of local-ci runner that does deterministic builds from a dependency graph. No changes, no build, massive parallelism if you want it

weird-eye-issue 4 hours ago | parent | prev | next [-]

They don't always have a great concept of time so for something like running a full test suite that takes a long time you should just tell it not to do that

4 hours ago | parent | prev | next [-]
[deleted]
djmips 3 hours ago | parent | prev [-]

have you ever worked for a big company where that's the status quo for any tiny change... hours on _full test suites_ over and over again.

notduckrabbit 3 hours ago | parent [-]

Astra will initiate test suites, find one more thing independently while its running, reinitiate complete test suite after fixing it, then find one more thing, then test again. Easy to burn through GH actions minutes if you're not careful orchestrating.