The difference here is that the scientific papers he downloaded weren't freely available to the public, like those scraped webpages would be. Corporate scrapers have been sued[0] in the past for scraping pages from behind a login page / paywall.

[0]: https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn