I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like:

- Spend most time prioritizing/discussing what to do.

- Once that's agreed, use Fable 5 High + 5.6 Sol XHigh come up with a design + plan. Agree on the high level plan. (Usually this just comes down to choosing where the change belongs on the spectrum between minimal patch <-> full redesign)

- Use Opus 5 or Sol Med to execute

- Auto-fix bugs and CI until green + thermonuclear review skill x3.

- Manual interrogation of change/nits

- Come up with QA plan and have Codex Computer Use execute on it

- Manually spot check the final result (usually a sizable diff, thousands of lines, complete feature E2E, etc)

I probably spend like $80 a day at least but I produce the output of 3 or 4 2022 engineers and probably at better quality. So it's easily worth it. Would I save money by switching to GLM 5.2 and such...perhaps? IDK. At our scale it's not worth the time spent building the eval harness to actually understand the performance tradeoff.

In our team's experience, the product of agents is generally The Homer (1). It does work, but it's vastly overengineered.

When I personally want tight code, I have to spend a considerable amount of time adjusting it manually:

- It needs to be trimmed down. In my experience, at least one agent I use struggles to produce minimalist designs, and it's very frustrating

- I need to consider whether there are solutions based on higher-level assumptions, that AIs typically miss

- I need to check whether there are off-the-shelf solutions - AIs like to reinvent the wheel

IMO, software production has become a mass-produced commodity in every sense - it's much more expensive to produce software manually, but the quality is not the same.

(1) https://simpsons.fandom.com/wiki/The_Homer

IME this works until it does not. This approach works well at the beginning of a greenfield project, but at the same time because it is so easy to add features, you will likely ship something that is way too over engineered. And that complexity will not amortize over next increments and will more likely lead to the entire project being a black box only fully understood by AI. However a more careful use of AI for targeted surgical changes is far more ”productive” in the long term IMO.

This is very close to my workflow but you forgot one important step:

- Suggest a better approach that makes the AI say, “That’s much simpler. And you’re right. My original plan was over-engineered.”

Do you have issues with performance at the moment? Right now I tend to find that it produces absolutely terrible design patterns and especially performance. I mean maybe I don't know exactly what area you're looking at but yeah for us we tend to find it's terrible wrt dB/caching/scaling and often any performance improvements it proposes end up actually shooting itself in the foot and being worse than before but it's not very good at testing in an organized way to even notice it made it worse despite repeated prompts to do so I mean if I prompt it to test performance in a handheld structured way (it is very bad at finding out what performance to test and why) before making changes I can usually figure it out but it usually takes insistence on the specifics to really ensure a good solution that will actually fix the problem

Output of 3 or 4 2022 engineers? Its that your self assessment? Output as in number of lines of code?

> Spend most time prioritizing/discussing what to do.

you should probably be doing this discussion work along with Fable 5. It will give good feedback if you're working on the correct things.

> Come up with QA plan and have Codex Computer Use execute on it

QA plan should be part of the above "design + plan", not after it. The implementer needs to be able to fully test before publishing a PR. This is true whether humans or agents are writing the code.

> Manually spot check the final result (usually a sizable diff, thousands of lines, complete feature E2E, etc)

unfortunately this is not really scalable with amount of code agents can produce, so you need independent (fresh context) agent reviewers to help. Ideally they only escalate to a human when really stuck.

> I probably spend like $80 a day at least

at a small startup you should be on the $200/month plan(s).

$80 sounds extremely low for what you're describing - are you on API token plans?

I have had some $3,000 token days - even without Fable. I don't see how this is sustainable.

My personal 20x plans get so much usage for so cheap. The consumer subsidies are crazy, but alas I can't use them for work.

Only spending $80 a day on Opus 5/Fable 5/GPT 5.6 Sol feels very low. I'll roll through a couple hundred dollars worth of credits a day with those models, the vast majority of which would be on non-coding tasks, and it's still a huge cost savings over me or my team having to do these things manually, if we'd even be able to do them at all.

But that's also why it's now easy to justify the cost of an Nvidia or Intel inference server with Kimi K3 locked and loaded :)

I was just about to say, how could routing possibly be worth it at the risk that the work output is sub par?

My experience is that your description works for a certain time, since you're knowledgeable of the codebase and can guide it. But after too many iterations with not hand-holding the llm, it quickly gets unwieldy.

The output yes, but do you produce the impact and value of 3 engineers? I have seen this workflow being toyed with too, and I find it to produce massively overengineered stuff that actual people don't really wanna use

>Auto-fix bugs and CI until green + thermonuclear review skill x3.

Gotta love this loop, I have it running while I'm asleep all the time.

Do you have tips for generating clean productive output per dollar?

> essentially unlimited AI spend budget

> I probably spend like $80 a day

This doesn’t sound like “unlimited”, I spend more than this out of pocket per day and I have a strict budget.

do the same across 20 terminals (as you should) and now you are up to $1.6k/day. would that give you output or 60-80 engineers? not a chance, right?

no one’s AI spent will be in question working a single terminal with carefully planned out and executed process you do

but I produce the output of 3 or 4 2022 engineers and probably at better quality.

Possibly, but the output of a 2022 engineer is about 1/10th of the output of a 2010 engineer, so it's an extremely low bar.

[dead]