| ▲ | throwaway290 3 hours ago | |||||||
what stops Kagi from indexing internet and makes them pay some guys to scrape search results from Google? one guy at Marginalia can do it and entire dev team at a PAID search engine can't? | ||||||||
| ▲ | DangitBobby 30 minutes ago | parent [-] | |||||||
As we've seen here on HN on the AI boom, it's not wonderful when a bunch of companies all use bots to scrape the entire web. Many sites only allow Google scrapers in robots.txt and the public will fight you hard if you scrape them without permission. It's just one of those things where it would be better for everyone if search engines could pay for access to the work that's done only once. | ||||||||
| ||||||||