"LLM-assisted code review won't be a huge disruptor" is quite the prediction. Because it already is in a very big way. The take on LLMs seems incredibly out of date and out of touch with reality. (Which of course, has moved/advanced very fast the past months/year)

▲

layer8 6 hours ago | parent | next [-]

He explains how he thinks it won’t be a disruptor:

> Just on that repeated experience, I suspect we have already seen more than half of the “worst software bugs found with LLM-tools” list.

On the other hand, it’s not clear to me how you think that it already is “in a very big way”.

	▲	ang_cire 4 hours ago \| parent [-]
		Which is such a crazy assertion, given that not even Mythos was trained explicitly as an exploit finder. Even if the US's profit-driven capital for AI dev dries up, China's state funding for results-driven AI isn't drying up, and they're going to keep building better models as long as the results are there (which they are), for better or worse.

▲

dgellow 6 hours ago | parent | prev | next [-]

That’s not his prediction, he made it clear they are useful. His conclusion for that section is this:

> The only real question for me is: Are the LLM-code-review tools economically viable outside the bubble?

▲

simianthoughts 6 hours ago | parent [-]

So is he suggesting that in the future, models of Opus 4.8 tier will no longer be as affordable? Like it would become 3-5x more expensive?

Now that's quite a prediction.

	▲	intended an hour ago \| parent \| next [-]
		I believe the added point was that new tools identify a group of bugs, and then there aren’t more bugs to find. At which point it becomes another tool you use and don’t think about. Also, from what I recall, Anthropic and OpenAI subsidize their prices by ~40x? That said, given the prices*quality of recently released open models, I think the cost issue is moot.
	▲	dgellow 5 hours ago \| parent \| prev \| next [-]
		I’m not exactly sure but I think he’s implying the AI labs aren’t profitable on inference? And that once the bubble pops those models won’t be available anymore? Just my assumptions
	▲	ai_fry_ur_brain 5 hours ago \| parent \| prev [-]
		[dead]

▲

delusional 6 hours ago | parent | prev | next [-]

Did you continue reading? His argument is exactly that, like all the previous model checkers, LLM's are going to give us some bugs for a while. Then that is going to stop.

His argument is not that they aren't going to find any bugs, but rather that at some point those bugs will be fixed. At which point we will continue on as usual.

▲

ang_cire 4 hours ago | parent | next [-]

How would they stop? Are you insinuating they're going to have reviewed all code at some point, and that new code for them to review will just cease to exist? Or that they'll just decide to stop finding bugs they're finding now in new code when they review it? All the previous model checkers didn't stop giving us bugs to fix, which is why his premise is wrong; every big company is still running SonarQube because it still gives you bug findings. So will AI.

If the argument were instead that it will cease to find new classes of vulnerabilities and bugs, that may very well be true, because that is a question of the limitations of programming and at a lower level computer architecture, but that's not the argument the author made.

▲

simianwords 6 hours ago | parent | prev | next [-]

> At which point we will continue on as usual.

This part is the load bearing claim. Why would you continue on as usual? I'm using LLM's everyday on code reviews and they still catch bugs.

▲

layer8 6 hours ago | parent [-]

The question is whether the balance of white-hat and black-hat LLM bug findings gives us a different enough result from the previous balance of white-hat and black-hat human bug findings to constitute a major disruption.

▲

simianwords 6 hours ago | parent [-]

I don't understand your point. How do you make your future code resistant to bugs without LLMs?

	▲	layer8 5 hours ago \| parent [-]
		That wasn’t the question. The question is how LLM code review is a huge disruptor. What is it disrupting? At best, with regard to code review LLMs just solve the problems that LLMs create.

▲

refulgentis 6 hours ago | parent | prev [-]

I did, his argument is that we've already discovered ~50% of all bugs discoverable by LLMs.

I'm treading lightly after you said "did you read it" to OP, I do believe we both understand that argument isn't nearly air-tight. (i.e. it implies either humans get so good at code that bug-introduction-rate falls percipitously, or, LLMs are so awesome they write all of our code bug-free. Neither of which jives with the thesis, that LLM code review is a nothingburger long term)

The best steelman we could say is "he meant 50% of all existing bugs in all currently existing code", which is still incompatible with a time-bound on their usefulness, unless we expect the rate of new code to fall percipitously.

The steelman I'm using, is they're speaking both loosely and strongly and intend us to understand these are strong opinions, held loosely, and they care for us enough to share.

▲

readthenotes1 6 hours ago | parent | prev [-]

Maybe he's trying to prove the point of why he's ready to retire...