You don't necessarily need JS, you just need something that can detect if Anybis is used and complete the challenge.

Sure, doesn't change anything though; you still need to spend energy on a bunch of hash calculations.

But then you rate limit that challenge.

You could setup a system for parellelizing the creation of these Anubis PoW cookies independent of the crawling logic. That would probably work, but it's a pretty heavy lift compared to 'just run a browser with JavaScript'.