Not a lawyer, but my understanding is that crawling is a gray area. By putting that into their terms of service that most of the planet is agreeing to, they protect themselves in the case someone complains.

Also, this is about crawling and has little to do with the license of the contents.

Not a lawyer, but my understanding is that if a Big Tech company doesn't explicitly rule out using your data, they're using your data. And if they rule it out explicitly, they will eventually still use your data.