How do we (or can we) know adversarial attacks don't work on us?

That's not the point I'm making. So in context of vision we pattern match too, and you can see it in the hundreds of optical illusions online. Our visual system relies on built in shortcuts, and they break in different places than a neural network's do. Image models get fooled by things we would never notice, and we get fooled by things they handle fine. IMO that's good evidence that we and these models use different algorithms to solve similar problems. I think the same is probably true of LLM "reasoning", there's some overlap with how we think, but a lot of it likely works very differently under the hood. Good example of this is anthropic's interpretability research where they found claude doing addition in a way that's nothing like the carry the one method it describes when you ask how it did it.

We're pretty confident that adversarial attacks do work on us, actually—we have entire industries built around it. Some of them even start with "adver-"...

There are classes of adversarial attacks targeting them that don't work as well on us, and vice versa.