| ▲ | We found a division by zero bug in FFmpeg with a vibecoded fuzzer(code.ffmpeg.org) |
| 67 points by dclavijo 2 hours ago | 39 comments |
| |
|
| ▲ | dabinat 2 hours ago | parent | next [-] |
| It’s interesting how AI may both raise and lower the quality of software. It’s very easy to send an AI agent on an open-ended bug hunt, and if it wastes a bunch of time and effort and finds nothing, no big deal. Time is much more important for a human developer with a salary. |
| |
| ▲ | dmix 2 hours ago | parent | next [-] | | Finding the bugs with LLMs is easy. Reviewing the output, cleaning it up, and making sure it doesn't break something else is the hard part. | | |
| ▲ | nonethewiser 21 minutes ago | parent | next [-] | | No one can keep up with the volume of code AI produces. We wont stop using AI. We will use AI to check AI. Of course this is crazy, but it will also unlock pretty insane scaling and productivity and ultimately we will manage it on either end via requirements and tests. | | |
| ▲ | harambae 7 minutes ago | parent | next [-] | | It's mostly (not entirely, but mostly) finding security issues in old human-written code. It'll eventually start running out of those. From that standpoint, it's not a crazy setup security-wise. Maybe still crazy for development. | |
| ▲ | krona 15 minutes ago | parent | prev [-] | | You're suggesting that LLMs get better at fixing bugs/vulnerabilities, but at the same time stop getting better at finding them? What if this difference is inherent and essential? |
| |
| ▲ | hombre_fatal 25 minutes ago | parent | prev [-] | | The missing part of this is that verifying the bug with LLMs is also easy, and so is adversarially reviewing the proposed fix with LLMs. The only thing left for you to do should be directional decisions. The LLMs should pause and rope you in if the fix involves directional/invariant changes. |
| |
| ▲ | evenhash an hour ago | parent | prev | next [-] | | > It’s very easy to send an AI agent on an open-ended bug hunt, and if it wastes a bunch of time and effort and finds nothing, no big deal. No big deal? It’s not like it’s free… tokens cost money. | | | |
| ▲ | Supermancho 2 hours ago | parent | prev | next [-] | | I don't care if you call it an over-engineered looping machine or what, there are concrete benefits to using LLMs for this. They work faster than developing your own looping algorithm and more often produce useful results than not. | |
| ▲ | eviks 2 hours ago | parent | prev | next [-] | | But what's your expectation of the net? | |
| ▲ | shevy-java an hour ago | parent | prev [-] | | I dislike AI, but if AI finds real bugs then this is in my opinion objectively a positive thing. Of course the question is what constitutes a real bug. | | |
| ▲ | hn_submit 8 minutes ago | parent | next [-] | | A.I. is useful for this. But it would be even more useful if all new code were written in Rust or some other memory-safe language. A.I. could also be used to port C/C++ codebases to Rust, which isn't economically feasible at the moment. | | | |
| ▲ | pixl97 an hour ago | parent | prev [-] | | Unfiltered models will help build exploits for the bugs they find, so there is some means of measuring their efficacy. | | |
| ▲ | klipt 4 minutes ago | parent [-] | | If you're just talking about security bugs. There are also non security bugs that don't have exploits but just make the user experience worse. |
|
|
|
|
| ▲ | ks2048 an hour ago | parent | prev | next [-] |
| No doubt fuzzers (vibecoded or otherwise) can be powerful, but can't you just mark all "/" as potential divide by zero errors? I guess sometimes developers think they "know" some variable won't be zero, but unless it checked explicitly or by the compiler, that shouldn't be trusted. |
| |
| ▲ | Someone 41 minutes ago | parent | next [-] | | > but can't you just mark all "/" as potential divide by zero errors? If you’re accepting large false positives rates: yes. If you want users to take your warnings serious: no. (Nitpick: you certainly don’t want to flag _all_ of them. Divisions by non-zero constants definitely should be excluded, for example (integer division by -1 can lead to overflow, but that would be a different warning)) | |
| ▲ | saghm 38 minutes ago | parent | prev | next [-] | | Fuzzers find inputs, not just "potential" errors that aren't triggerable. | |
| ▲ | dooglius an hour ago | parent | prev | next [-] | | What are you suggesting and how would it be different than how SIGFPE already works? | |
| ▲ | wvbdmp an hour ago | parent | prev [-] | | I mean there could be a guard clause? But yeah, seems like this could be statically evaluated like how some IDEs see a null check and don’t complain about nullability within the same scope. |
|
|
| ▲ | robertlagrant 40 minutes ago | parent | prev | next [-] |
| What we need is a numeric type that cannot be zero. |
| |
| ▲ | yeputons 5 minutes ago | parent | next [-] | | And also cannot be INT_MIN, otherwise -1 / INT_MIN is undefined behaviour(!) in C and C++. | |
| ▲ | rhdunn 11 minutes ago | parent | prev | next [-] | | It would be more flexible for a compiler to reuse the range analysis logic used in optimizations for statically verifiable divide by zeros. That way you could extend it to other things like statically verifiable overflows. | |
| ▲ | drdaeman 30 minutes ago | parent | prev [-] | | What we need are refinement types, where there’s a base type and a predicate. F* has this: val (/) : int -> (divisor:int { divisor <> 0 }) -> int
|
|
|
| ▲ | 12j3afAv an hour ago | parent | prev | next [-] |
| Generating an incorrect input file seems to be the easiest task of all for any fuzzer. Generating correct input to get deep into the call stack and then finding something is the hard part. |
|
| ▲ | akshay_akula an hour ago | parent | prev | next [-] |
| The open ended bug hunt is the best use case for these agents. Finding nothing costs a few dollars, finding a division by zero in ffmpeg pays for itself. |
| |
| ▲ | wy35 17 minutes ago | parent [-] | | Unrelated to the submitted link -- just checked your comment history and all of your comments are AI-generated like this one. What's the motivation for this? | | |
| ▲ | f311a a minute ago | parent [-] | | He won’t reply, he’s busy promoting himself and his peojects with AI. |
|
|
|
| ▲ | Surac 2 hours ago | parent | prev | next [-] |
| send patches |
| |
|
| ▲ | VCFundedGenYer 2 hours ago | parent | prev [-] |
| The fruits of using LLMs to code.
You'll waste far more time finding what it quietly and subtly wrecked than you would have if you just coded it yourself. |
| |
