I assume this is for people who don’t want their sites crawled and ingested for AI training. With the goal of making it cost-prohibitive to do so.

Of course, one side effect will be crippling search engines as well. Maybe less important for driving traffic in the social media era?

People are trying to separate search crawling from AI crawling but Google is strongly resisting. I don't know how it will play out.