| ▲ | ltbarcly3 4 hours ago | |
They seem to conclude Luna is a better value, but their analysis is dumb. They just break it down to $/bug found. However, Luna missed 23 bugs that Astra found, and identified 24 bugs that weren't really bugs. That's horrible. Astra had 96% precision. The cost to care about here isn't just how much it costs to run the code review, or the cost per true-positive. It's the cost of dealing with this system. A code review system that is right about 2/3 sucks, and one that misses another 1/3 of the bugs is also a lot worse. The Astra code review quoted here would become the foundation of how the team works, the Luna version is at best helpful to find some stuff but does not dramatically increase your confidence. It also will force humans or better AI's to have to run down a lot of false positives, and that is treated as free here. Actual conclusion: The cost for Astra is low in absolute terms compared to the cost of bugs and human attention, and the added value is far far more than the added cost. | ||
| ▲ | samusiam 31 minutes ago | parent [-] | |
To me, one interesting piece of analysis was whether Luna had benefit on top of Astra -- i.e., running both and synthesizing their findings. But even with that, it raises the question whether running a second Astra pass, or Sol, or even a model from another family (GLM? Fable?) would deliver even more additive benefit. | ||