Do you have issues with performance at the moment? Right now I tend to find that it produces absolutely terrible design patterns and especially performance. I mean maybe I don't know exactly what area you're looking at but yeah for us we tend to find it's terrible wrt dB/caching/scaling and often any performance improvements it proposes end up actually shooting itself in the foot and being worse than before but it's not very good at testing in an organized way to even notice it made it worse despite repeated prompts to do so I mean if I prompt it to test performance in a handheld structured way (it is very bad at finding out what performance to test and why) before making changes I can usually figure it out but it usually takes insistence on the specifics to really ensure a good solution that will actually fix the problem

▲

extr 6 hours ago | parent | next [-]

Performance is better than ever. It's never been more practical to set up wildly complex synthetic test environments and measure perf wins. Plus the models will find every possible algorithmic/design improvement.

It actually gives me quite an uncanny feeling, bulldozing over years of human optimization work with a newer, "perfect" design. Like bringing an AK-47 back to the middle ages.

▲

RugnirViking 4 hours ago | parent | next [-]

> the models will find every possible algorithmic/design improvement

it's so hard to square such totalizing statements with my day to day experience with fable and sol, (every possible, improvement, really?? they are NOT omniscient) arguing with them/my colleagues' agents that no they have slowed down the system 200x with their terrible change, doing string operations on millions of db rows, trying to get it to understand that I don't care that it's calling it a "cache" if a cache hit is slower than what we had before.

These agents do let you learn codebases quickly, and produce code way faster. I don't look at IDEs all that often. But literally multiple times every single day I catch them doing something stupid.

I don't think its impossible that we could get better performance from the agents. I know ive tried all sorts of workflows and skills, few of which seem to have much effect on the things the models struggle with. I think a big part of it is encoding enough context for large codebases, and providing it with all the tools it needs to make it successful, things to automatically check its work, etc. But that's not automatic, in fact its generally a terrible judge of what it needs or what its bad at

	▲	extr 3 hours ago \| parent [-]
		yeah it's true, you do have to guide them. i find that the key is you have to know what's possible. you have to have the instinct for "this really shouldn't be so difficult". my junior SWE coworkers have the same trouble as your coworkers. but the revolution is it doesn't take that long. in like 15 minutes you can chat with fable and get to the meat of whatever the issue is with repeated questioning. and then it does the solution for you. so it's not magic but it's still like a 100x speedup.

▲

app13 6 hours ago | parent | prev [-]

I needed to thoroughly test rerankers on my companies rather unique corpus.

Opus and I wrote a parallelized test harness and labeled groundtruth in around 2 hours.

In 2022 that would've likely been all I did for a couple sprints

	▲	steve_adams_86 5 hours ago \| parent \| next [-]
		I encounter this regularly and it still feels weird. That sense that you did something better in a few days than you would have in a month 5 years ago. It's like buying a table saw for wood working. One crazy thing I think about often is how there are so many correctness and testing harnesses that would have taken weeks to build in the past so we simply never would have. We'd just do our best then wait and see what comes to the surface. This is a huge part of what makes it possible to actually make better software with LLMs in my opinion. It isn't just 'LLM codes better than I ever could' (that's often untrue still) but 'LLM enables me to make assertions about the program to degrees that would have been absurdly impractical in the past'. It's huge
	▲	extr 6 hours ago \| parent \| prev [-]
		Yes 100%. This morning I casually prompted Codex to drive the browser to complete extensive performance testing in-situ that would have literally been weeks of work before. Probably in reality it just wouldn't have been done, and performance guarantees would have been attempted up front via more careful design. In this case the design was also AI generated, and there were limited wins to be found because the design was already superb.

▲

manmal 6 hours ago | parent | prev [-]

Are you using the SOTA models at very high reasoning during planning? IME that makes a LOT of a difference. I‘d also never let them just rip into the architecture, but always push back and ask for alternatives first. Once the overall plan is nailed, not that much can go wrong. Provided it’s a reasonable change set and not a 20k LOC PR.

	▲	RugnirViking 5 hours ago \| parent [-]
		fable or sol w/ very high both planning and execution, yeah. I feel the "push back" part is a big part of my job now (on every step, planning, execution, and review) yeah, but that feels pretty incompatible with the sorts of "just let it do what it wants" which other people seem to be claiming