| ▲ | deepsun 4 hours ago |
| BigQuery has "public datasets", so users can even run complex SQL on it, but it's them who pays for it, not you. You only pay for data storage. |
|
| ▲ | primitivesuave 39 minutes ago | parent | next [-] |
| Thanks for that tip, this is exactly how I would do this if I had to do it from scratch. Just in case it's useful for anyone else: https://docs.cloud.google.com/bigquery/public-data |
|
| ▲ | dpoloncsak 4 hours ago | parent | prev | next [-] |
| The crawlers would have still just hammered their site though, right? |
| |
| ▲ | primitivesuave 38 minutes ago | parent | next [-] | | Yes, if I wanted to put a nice HTML interface over the query results then I would still end up with the same problem where some combinatoric explosion of query parameters to the `/search` endpoint, most of which are cache misses, leads to many many TBs of network egress. | |
| ▲ | pavel_lishin 3 hours ago | parent | prev [-] | | I think the idea is that they could store the data in BigQuery, and point users of the site there. | | |
| ▲ | dpoloncsak 3 hours ago | parent | next [-] | | Sure, but now you're moving the site from "Free data presented in a pleasant way to view" to a "pay-as-you-go database". Your audience shifts dramatically, and you lose the ability to share the data you're trying to present. | | | |
| ▲ | tekne 29 minutes ago | parent | prev | next [-] | | There is also the deep magic... https://github.com/phiresky/sql.js-httpvfs | |
| ▲ | PunchyHamster 3 hours ago | parent | prev [-] | | The crawlers would have still just hammered their site though, right? | | |
| ▲ | 40four 3 hours ago | parent [-] | | Why is your comment exactly word for word of another comment just one level above in the comment chain? | | |
| ▲ | gorgonian 3 hours ago | parent | next [-] | | Maybe because they restated what they said instead of addressing to the previous commenter’s point. | |
| ▲ | taneq 3 hours ago | parent | prev [-] | | Because the crawlers would still have hammered their site, though. (The GP post doesn’t actually meaningfully address the issue being raised. Adding BigQuery or whatever would not change the fact that (a) they already offered a method of getting all of the data in a cost effective way, and (b) the issue was that the crawlers hammered the site hard enough to make it economically unviable.) |
|
|
|
|
|
| ▲ | lokar 3 hours ago | parent | prev [-] |
| I think snowflake has similar, you can rent it out or make it free |
