In 2015, I was able to get to 1 million read/write queries per second on only a couple nodes and tested this with multiple databases, it required (at the time) decent network tuning and node placement inside AWS but it cost me about 10 to 15 dollars per run if I recall correctly, obviously there is the matter of scaling such performance and so I want to recognize the engineering effort gone into this but this is too much money. This reminds of when one of my teams used Hadoop to process only a a few terabytes of offline data and were able to process the WHOLE THING in only a few hours. I did not have the heart or courage to tell them during the demo that this was overkill, but I did write a very simple (and small) piece of code that could extract all the signals from the offline files in mere seconds with careful network planning and storage optimization and invited them for a demo/lunch and learn next week.

▲

tomnipotent 6 hours ago | parent | next [-]

> all the signals from the offline files in mere seconds

That sounds unlikely unless those machines had access to crazy disk I/O. A local RAID 5/10 with 8 drives would still take 30-50 minutes just to read that much data. Even with a mid-range SAN you still would have spent 15-25 minutes just reading data. This assumes 7.2K SAS/SATA since SSD/NVMe were not ubiquitous in 2015, but even with 2015-era SSDs you're still looking at half that much time spent reading.

▲

farazbabar 3 hours ago | parent | next [-]

Lots of EBS volumes mounted via 25GBPs or higher network, positioned carefully onto a single rack where possible and carefully tuned network/ip/os for both clients and servers. It is not feasible in most deployments as this would not have scaled to real production loads (RAID configurations requiring redundancy alone would slow you down, not to mention costs of using that many EBS volumes on extra high network/IO/provisioned IOPS nodes). This was only meant to prove what was possible in AWS at the time.

	▲	tomnipotent 2 hours ago \| parent [-]
		Not in 2015. 25GBps didn't even arrive until 2016, and dedicated EBS bandwidth didn't arrive until 2017.

▲

jeffbee 3 hours ago | parent | prev [-]

Your statement about SSD availability and performance does not fit with my personal recollections of history. Smoking fast SSDs were completely vanilla items you could order from anyone in 2015. For example, the Intel DC P35/6/700 series. Each of these was as good as 25 HDDs for linear reads and more like 5000 HDDs for random reads. The U.2 specification for putting SSDs in boxes was formalized in 2011. Outside of the mass market, cloud builders had private SSD designs long before 2015.

	▲	tomnipotent an hour ago \| parent [-]
		Less than 5% of commercial storage capacity was SSD of any flavor in 2015 (NVMe probably less than 1%), and the price-per-GB was anywhere from 5-15x. NVMe hardware RAID didn't happen until 2017, and AWS SSD via EBS was capped at 320 MB/s (~500 MB/s with striped EBS volumes). Before 2015 it was capped at 160 MB/s. In practice even NVMe SSDs at the time were maybe 10x faster for sequential R/W and maybe only 200-500x for random I/O. Most databases in 2015 were limited by serial dependency chains like walking a B-tree to figure out what nodes to read next and couldn't max out what an NVMe was capable of. Not to mention we were stuck with libaio in 2015 that made O_DIRECT ops synchronous. So the OPs claim of reading multiple terabytes of data in seconds on AWS in 2015 literally wasn't technically possible. SSDs simply were not as ubiquitous in 2015 as they are today, and I didn't know anyone using them outside of mission critical systems were the cost made sense.

▲

kinduff 4 hours ago | parent | prev [-]

What happened next?

	▲	farazbabar 3 hours ago \| parent [-]
		That's a good question, I was very high up in the leadership, couple levels below the CIOs and the biggest surprise for the team was that I could write even a single line of code. In large organizations, changing direction is not always easy so I could not get them off of using a data platform, which to be honest was acceptable because we could not ask the team to write such custom code for each job going forward, I did manage to shift them towards flink which I think was better suited to our actual real-time needs.