SSL Heartbleed is a good example. Or pretty much any vulnerability that needs understanding of how memset or malloc works, or anything where you have to use leaky functions to create a specific offset because that's where the return (eip) in assembly is, so that you can modify/exploit that jmp or cmp call.

These kind of things are very hard for LLMs because they tend to forget way too much important information about both the code (in the branching sense) and the program (in the memory sense).

I can't provide a schematic for this, but it's pretty common in binary exploitation CTF events, and kind of mandatory knowledge about exploit development.

I listed some nice CTFs we did with our group in case you wanna know more about these things [1]. I think in regards to LLMs and this bypass/sidechannel attacks topic I'd refer to the Fusion CTF [2] specifically, because it covers a lot of examples.

[1] https://cookie.engineer/about/writeups.html

[2] https://exploit.education/fusion/

▲

tptacek a day ago | parent [-]

Wait, I don't understand why Heartbleed is at all hard for an agent loop to uncover. There's a pattern for these attacks (we found one in nginx in the ordinary course of a web app pentest at Matasano --- and we didn't find it based on code, though I don't concede that an LLM would have a hard time uncovering these kinds of issues in code either).

I think people are coming to this with the idea that a pentesting agent is pulling all its knowledge of vulnerabilities and testing patterns out of its model weights. No. The whole idea of a pentesting agent is that the agent code --- human-mediated code that governs the LLM --- encodes a large amount of knowledge about how attacks work.

▲

cookiengineer a day ago | parent [-]

I think I'd differ between source code audits (where LLMs already are pretty good at spotting bugs if you can convince them to) and exploit development here.

The former is automated by a large part already with fuzz testing of all kinds, so you wouldn't need an LLM if you knew what you were doing and have a TDD workflow or similar that checks against memleaks (say, with valgrind or similar approaches).

The latter part is what I was referring to where I had hope initially that DNCs could help with that, and what I'd say that right now LLMs cannot discover this, only repeat and translate it (e.g. similar vulnerabilities in the past discovered by humans in another programming language).

I'm talking specifically about discovery here because transformers lose symbolic inference, and that's why you can't use them for exploit generation. At least I wasn't able to make them work for the DARPA challenges, and had to use an AlphaGo based model combined with a CPPN and some techniques that worked in ES/HyperNEAT.

I suppose what I'm trying to say is that there's a missing understanding of memory and time when it comes to LLMs. And that is usually manually encoded/governed how you put it by humans. And I would not count that as an LLM doing it, because you could have just automated the tool use without an LLM and get identical results. (When thinking e.g. about an MCP for kernel memory maps or say, valgrind or AFL etc)

▲

tptacek a day ago | parent [-]

We're talking about different things here. A pentesting agent directly tests running systems. It's a (much) smarter version of Burp Scanner. It's going to find memory disclosure vulnerabilities the same way pentesters do, by stimulus/response testing. You can do code/test fusion to guide stimulus/response, which will make them more efficient, but the limiting factor here isn't whether transformers lose symbolic inference.

Remember, the competition here is against human penetration testers. Humans are extremely lossy testing agents!

If the threshold you're setting is "LLMs can eradicate memory disclosure bugs by statically analyzing codebases to the point of excluding those vulnerabilities as valid propositions", no, of course that isn't going to happen. But nothing on the table today can do that either! That's not the right metric.

	▲	cookiengineer a day ago \| parent [-]
		> Humans are extremely lossy testing agents! Ha, I laughed at that one. I suppose you're right :D