> What we've found is that giving LLM security agents access to good tools (Semgrep, CodeQL, etc.) makes them significantly better

100% agree - I spun out an internal tool I've been using to close the loop with website audits (more focus on website sec + perf + seo etc. rather than appsec) in agents and the results so far have been remarkable:

https://squirrelscan.com/

Human written rules with an agent step that dynamically updates config to squash false positives (with verification) and find issues while also allowing the llm to reason.