| ▲ | Benchmarking coding agents on Databricks' multi-million line codebase(databricks.com) |
| 65 points by tanelpoder 10 hours ago | 21 comments |
| |
|
| ▲ | redmalang 3 hours ago | parent | next [-] |
| We have an internal proxy (that I've been meaning to open source for ages) that routes all llm usage at our company, which allows us to see data in realtime. Its been fascinating how rapidly Pi has been adopted. Moreover since its pretty hackable, we've been able to automatically aggregate context from pi sessions, which has resulted in Pi efficacy being higher as more people use it, putting in place a interesting virtuous loop.
I didn't expect this outcome: for whatever reason I assumed proprietary harnesses fine tuned to work with a companies' models would work better?
ps/random aside: there is something slightly off about Pi's edit command, we are planning to investigate this further and patch this as we have quite a few session traces now.. |
| |
|
| ▲ | lukax 2 hours ago | parent | prev | next [-] |
| Could it be that users of Pi are more senior and know better how to prompt and that's why the pass rate is higher? |
|
| ▲ | cpard 4 hours ago | parent | prev | next [-] |
| This was mostly because Sonnet 5 worked longer and read more to get there, consuming 1.9x more tokens. I have experienced similar behavior between opus and haiku when benchmarking Dara engineering tasks. The “cheaper” model takes many more turns to figure out the task and this is without taking into account other important factors. Another interesting behavior that I observed is that Haiku tended to cheat more maybe because it was having a harder time to find the root cause of the problem. Benchmarking and evaluation of agentic systems is very interesting and if there’s one thing that someone should keep from the Databricks post is how important is for everyone to build and run their own. |
|
| ▲ | yodon 4 hours ago | parent | prev | next [-] |
| I wish they'd do a follow-on post drilling into the impact of the programming language on cost-per-task, specifically looking at cost to complete tasks in mainstream strongly typed languages (eg. C#, TypeScript) vs dynamic languages (eg. Python, JavaScript). Does the additional verbosity of the language help or hurt cost per task? |
| |
| ▲ | trollbridge an hour ago | parent [-] | | I don't have hard data, but we have shifted to Rust and Swift (for frontend UI) for the bulk of our dev simply because it is a lot more predictable, easier for tool calls to edit, the build steps produce easier output for the agent to loop on, the tests are easier to write/get results from, etc., although I am mostly measuring this in time, not cost. Once the thing is rock-solid it's relatively easy to do a Swift->HTTP/HTML/CSS/React/TypeScript conversion. |
|
|
| ▲ | falaki 7 hours ago | parent | prev | next [-] |
| 1) Many models are now competitive at the top tier, including open source.
2) GLM 5.2 in particular was a major step forward in open source coding agent performance,
3) Harnesses make a huge difference in cost-performance.
4) Cheaper per-token does not imply cheaper per-task. |
| |
| ▲ | falaki 7 hours ago | parent [-] | | Also they suggest every company should build their own benchmark and repeat these tests with new models instead of relying on the SWE bench. | | |
| ▲ | fmind-dev 34 minutes ago | parent [-] | | It takes time and effort to build such benchmark. It works at Databricks scale, I'm not sure smaller companies are ready to invest on internal benchmarks. But they are more vendor neutral, now they don't sell their own model. It's interesting from a benchmark point of view. |
|
|
|
| ▲ | zkmon 3 hours ago | parent | prev [-] |
| > Databricks’ multi-million line codebase The combined size of codebases for the underlying opensource products (Apache Spark etc) might be around 1M lines, I think. Why does the orchestration/management layer, that is "databricks", exceed the sizes of the core products? |
| |
| ▲ | trollbridge an hour ago | parent | next [-] | | Because if they're like previous places I worked out with proprietary commercial codebases, the code is old, has been around a long time, and trying to shrink it is not a priority. It's a good stress test for the LLM because it is not an "ideal" codebase. | |
| ▲ | alexott 27 minutes ago | parent | prev | next [-] | | :facepalm: "orchestration/management layer" that just runs dozens of millions of VMs per day, across the three clouds, with things like model serving, AI gateway, OLTP (Lakebase/Neon), Unity Catalog, etc. Forget Databricks == Apache Spark... | |
| ▲ | ozgrakkurt 2 hours ago | parent | prev | next [-] | | Old codebase, you always add code and never remove it. So it is expected to be like this. Deleting code is difficult and almost never makes sense afaik | | |
| ▲ | wwind123 an hour ago | parent | next [-] | | On one hand, I understand that some old code is hard to delete because it's hard to detangle a lot of the legacy dependency. On the other hand, too much useless old code existing in the code base by itself could become a big maintenance burden for both humans and AI. In some cases at some point it might become more economical to just invest a bunch of resource to detangle the dependencies to be able to remove the old code. | |
| ▲ | tijs an hour ago | parent | prev [-] | | Every line you delete is a line you no longer need to maintain. We aggressively prune old code in our apps and it has definitely helped with maintainability. For a mobile app it’s also code you don’t ship so that’s a nice bonus which I guess is not much of an argument on a backend codebase |
| |
| ▲ | appplication 3 hours ago | parent | prev [-] | | LoC isn’t a super helpful metric so I think the better question is why is the headline using it. I can say I’ve personally created about 200k LoC code in the last 5 years and most of that has some value. But it really doesn’t say might about how much value or really anything else meaningful. | | |
| ▲ | InsideOutSanta 2 hours ago | parent [-] | | They're probably mentioning the size of the code base as an indicator for how difficult it is for an LLM to understand where and how to make changes. |
|
|
