| ▲ | morkee 8 hours ago | |
I hate to be a downer but... > UK planning data is technically public. it's public, but still copyrighted by those who submitted it the councils also have database rights over their database, unless you've obtained explicit permission from them directly https://en.wikipedia.org/wiki/Database_right#United_Kingdom > I ended up writing several scrapers: a standard requests-based one, a Playwright-based one for councils that block anything that doesn't look like a real browser, and a curl_cffi one for TLS fingerprinting. so they're explicitly trying to stop you doing this, and ... you're openly admitting to bypassing their technical measures to try and stop you? have you heard of the Computer Misuse Act? I doubt the 240 councils are going to be happy once they find out you've done this, especially if you're selling it on for profit | ||
| ▲ | mebkorea 8 hours ago | parent [-] | |
Fair points and I appreciate the feedback. Database right is real but the threshold is "substantial part". I'm literally only showing aggregates and letting people search by postcode. I'm not completely republishing council databases. Think that's defensible, but not gonna pretend that it's 100% black and white. On CMA, I'd push back. That's about unauthorised access. These portals are public-facing and the data's published deliberately for people to view. Rotating user-agents isn't bypassing security in any meaningful way... I'm not breaking auth or guessing passwords. I back off when portals signal they're unhappy (Liverpool's WAF actively rate-limited me which is why that data's stale). No council has reached out so far. Could change ofc. Solo founder with no legal team though, so happy to be told I've got it wrong. | ||