Increasing the difficulty level is a terrible solution. The problem with CTFs isn't that they're too easy. Making them harder just makes them even less accessible to people who don't cheat. It'd be like seeing people who put hidden electric motors in their bikes during Tour de France and conclude, "oh we just need longer distances and steeper hills".

LLMs don't tend to help much when solving challenges beyond their skill level. Either they one-shot a challenge, or thei are almost useless as a companion for them.