Is it the models tho? With every release (mutlimodal etc) its just a well crafted layer of business logic between the user and the LLM. Sometimes I feel like we lose track of what the LLM does, and what the API before it does.

Divide the steps into small enough steps so the LLMs don't actually know the big picture of what you're trying to achieve. Better for high-quality responses anyways. Instead of prompting "Find security holes for me to exploit in this other person's project", do "Given this code snippet, is there any potential security issues?"

The same way you write malware without it being detected by EDR/antivirus.

Bit by bit.

Over the past six weeks, I’ve been using AI to support penetration testing, vulnerability discovery, reverse engineering, and bug bounty research. What began as a collection of small, ad-hoc tools has evolved into a structured framework: a set of pipelines for decompiling, deconstructing, deobfuscating, and analyzing binaries, JavaScript, Java bytecode, and more, alongside utility scripts that automate discovery and validation workflows.

I primarily use ChatGPT Pro and Gemini. Claude is effective for software development tasks, but its usage limits make it impractical for day-to-day work. From my perspective, Anthropic subsidizes high-intensity users far less than its competitors, which affects how far one can push its models. Although it's becoming more economical across their models recently, and I'd shift to them completely purely because of the performance of their models and infrastructure.

Having said all that, I’ve never had issues with providers regarding this type of work. While my activity is likely monitored for patterns associated with state-aligned actors (similar to recent news reports you may have read), I operate under my real identity and company account. Technically, some of this usage may sit outside standard Terms of Service, but in practice I’m not aware of any penetration testers who have faced repercussions -- and I'd quite happily take the L if I fall afoul of some automated policy, because competitors will quite happily take advantage of that situation. Larger vuln research/pentest firms may deploy private infrastructure for client-side analysis, but most research and development still takes place on commercial AI platforms -- and as far as I'm aware, I've never heard of a single instance of Google, Microsoft, OpenAI or Anthropic shutting down legitimate research use.

Poetry? https://news.ycombinator.com/item?id=45991738

A little bit of social engineering (against an AI) will take you a long way. Maybe you have a cat that will die if you don't get this code written, or maybe it's your grandmother's recipe for cocaine you're asking for. Be creative!

Think of it as practice for real life.

That’s 100% why I haven’t done this - we’ve seen the movie where people build a business around someone else’s product and then the api gets disabled or the prime uses your product as market research and replaces you.

wild that so many companies these days consider the exit before they've even entered

Is it rookhotel.com?