|
| ▲ | saberience 7 hours ago | parent | next [-] |
| Well, at my old company we had some datasets in the 6-8 PB range, so tell me how we would run analytics on that dataset on an Intel NUC. Just because you don't have experience of these situations, it doesn't mean they don't exist. There's a reason Hadoop and Spark became synonymous with "big data." |
| |
| ▲ | dapperdrake 5 hours ago | parent | next [-] | | These situations are rare not difficult. The solutions are well known even to many non-programmers who actually have that problem: There are also sensor arrays that write 100,000 data points per millisecond. But again, that is a hardware problem not a software problem. | |
| ▲ | 7 hours ago | parent | prev [-] | | [deleted] |
|
|
| ▲ | literalAardvark 8 hours ago | parent | prev | next [-] |
| Well yeah, but that's a _very_ different engineering decision with different constraints, it's not fully apples to apples. Having materialised views increases insert load for every view, so if you want to slice your data in a way that wasn't predicted, or that would have increased ingress load beyond what you've got to spare, say, find all devices with a specific model and year+month because there's a dodgy lot, you'll really wish you were on a DB that can actually run that query instead of only being able to return your _precalculated_ results. |
|
| ▲ | DetroitThrow 7 hours ago | parent | prev [-] |
| >Datasets never become big enough… Not only is this a contrived non-comparison, but the statement itself is readily disproven by the limitations basically _everyone_ using single instance ClickHouse often run into if they actually have a large dataset. Spark and Hadoop have their place, maybe not in rinky dink startup land, but definitely in the world of petabyte and exabyte data processing. |
| |
