| ▲ | Property-Based Testing Caught a Security Bug I Never Would Have Found(kiro.dev) |
| 39 points by nslog 14 hours ago | 14 comments |
| |
|
| ▲ | Piraty 13 minutes ago | parent | next [-] |
| It's always good to write tests with the "The Enterprise Developer from Hell" in mind: https://fsharpforfunandprofit.com/posts/property-based-testi... |
|
| ▲ | philipwhiuk 3 hours ago | parent | prev | next [-] |
| > We write unit tests for the happy path, maybe a few edge cases we can imagine, but what about the inputs we'd never consider? Many times we assume that LLMs are handling these scenarios by default, Do we? |
| |
| ▲ | sevensor an hour ago | parent | next [-] | | What’s interesting to me about this, reckless as it is, is that the conversation has begun to shift toward balancing LLMs with rigorous methods. These people seem to be selling some kind of AI hype product backed by shoddy engineering, and even they are picking up on the vibe. I think this is a really promising sign for the future. | |
| ▲ | RGamma 3 hours ago | parent | prev [-] | | I've seen companies advertise with LLM generated claims (~Best company for X according to ChatGPT), I've seen (political) discussions being held with LLM opinions as "evidence". So it's pretty safe to say some (many?) attribute inappropriate credence to LLM outputs. It's eating our minds. |
|
|
| ▲ | mhitza 5 hours ago | parent | prev | next [-] |
| Technically a property based test caught the issue. What I've found surprising is that the __proto__ string is a fixed set from the strings sampling set. Whereas I'd have expected the function to return random strings in the range given. But maybe that's my biased expectation being introduced to property-based testing with random values. It also feels like a stretch to call this a property-based test, because what is the property "setters and getters that work"? Cause I expect that from all my classes. |
| |
| ▲ | arnsholt 2 hours ago | parent [-] | | Good PBT code doesn't simply generate values at random, they skew the distributions so that known problematic values are more likely to appear. In JS "__proto__" is a good candidate for strings as shown here, for floating point numbers you'll probably want skew towards generating stuff like infinities, nans, denormals, negative zero and so on. It'll depend on your exact domain. |
|
|
| ▲ | sublinear 5 hours ago | parent | prev | next [-] |
| > Is this exploitable? No. ... JSON.stringify knows to skip the __proto__ field. ... However, refactors to the code could ... [cause] subtle incorrectness and sharp edge cases in your code base. So what? This line of what-if reasoning is so annoying especially when it's analysis for a language like javascript. There's no vulnerability found here and most web developers are well aware of the risky parts of the language. This is almost as bad as all the insane false positives SAST scans dump on you. Oh I'm just waiting to get dogpiled by people who want to tell me web devs are dumber than them and couldn't possibly be competent at anything. |
| |
| ▲ | oncallthrow 2 hours ago | parent | next [-] | | > most web developers are well aware of the risky parts of the language In my experience this really isn’t true. Most web developers I know are not familiar (enough) with prototype pollution. By the way, this isn’t because they are “dumb”. It’s the tool’s fault, not the craftsman’s, in this case. Prototype pollution is complicated and surprising | |
| ▲ | yakshaving_jgt 3 hours ago | parent | prev | next [-] | | > most web developers are well aware of the risky parts of the language I don't think this is true, and I think that's supported by the success of JavaScript: The Good Parts. It would be unfair to characterise a lack of comprehensive knowledge of JavaScript foot-guns as general incompetence. | |
| ▲ | jgalt212 an hour ago | parent | prev [-] | | > insane false positives SAST scans dump on you Great LLM use case: Please explain to the box ticking person why these "insane false positives SAST" are false and / or of no consequence. |
|
|
| ▲ | mananaysiempre 6 hours ago | parent | prev [-] |
| TL;DR: obj[key] with user-controlled key == "__proto__" is a gift that keeps on giving; buy our AI tool that will write subtle vulnerabilities like that which you yourself won’t catch in review but then it will also write some property-based tests that maybe will |
| |
| ▲ | fireflash38 3 hours ago | parent [-] | | Don't forget you can use AI to turn a 50 word blog post into a 2,000 word one! | | |
| ▲ | mirthturtle 2 hours ago | parent | next [-] | | For real. The bullet-point summary at the beginning with a "Why this matters for..." immediately followed by, "This isn't just a theoretical exercise—it's a real example of..." Dead giveaways. | |
| ▲ | raphting 22 minutes ago | parent | prev [-] | | Exactly this!
AI fluff all over in that article. |
|
|
