It cost $250,000 to do this run but it feels worth it.

▲ jeffbee 8 hours ago | parent | next [-]

That's roughly 250x more than it would cost to perform this stunt using on-demand Cloud Bigtable, if my math checks out (~1150 nodes @ 85¢/hour for 1h).

▲ dataviz1000 7 hours ago | parent [-]

  > 512 shards, each with one Postgres primary each on an r8g.16xlarge
  > 480 Neki routers, each on its own 8xlarge instance
  > We sustained 118,538,803 QPS for 16 minutes across 512 shards and 1.22 PiB of data. Our largest recording was 118,747,267.

  Component                      Detail                           Monthly  Hourly  16-min burst
  ---------------------------------------------------------------------------------------------
  Shard compute                  512x r8g.16xlarge                 $1.41M  $1,930          $515
  Router compute                 480x r8g.8xlarge*                  $661K    $905          $241
  Storage (gp3 floor)            1.22 PiB @ $0.08/GB-mo             $102K    $140           $37
  Storage (io2 floor)            1.22 PiB @ $0.125/GB-mo            $160K    $219           $58
  IOPS (io2, light)              5K IOPS/shard, tiered rate         $166K    $228           $61
  IOPS (io2, medium)             20K IOPS/shard, tiered rate        $666K    $912          $243
  IOPS (io2, worst-case)         231,517 IOPS/shard (0% cache)     $4.56M  $6,251        $1,667
  ---------------------------------------------------------------------------------------------
  Total (gp3 floor)                                                $2.17M  $2,975          $793
  Total (io2 floor)                                                $2.23M  $3,054          $814
  Total (io2 + light IOPS)                                         $2.40M  $3,282          $875
  Total (io2 + medium IOPS)                                        $2.90M  $3,966        $1,058
  Total (io2 + worst-case IOPS)                                    $6.79M  $9,305        $2,481

▲

svuiv 6 hours ago | parent | next [-]

your calculation does not account for the load generation and cross-az network cost

▲

secondcoming 5 hours ago | parent | prev | next [-]

I see the machine sizes some people are allowed to use and I cry. Here I am being asked to downsize our VMs to only use 32 GB RAM.

▲

jeffbee 6 hours ago | parent | prev [-]

Sounds more right. Where did $250k come from?

▲

fizx 6 hours ago | parent [-]

I think Sam is an exec at the company.

Presumably it took more than one attempt, or there's some humanpower in the budget.

	▲	5 hours ago \| parent [-]
		[deleted]

▲ handfuloflight 8 hours ago | parent | prev | next [-]

I did not know men could build such things.

▲ whalesalad 8 hours ago | parent | prev [-]

I estimated the cluster to achieve this was ~$3-4k per-hour. I am thinking there is a typo on the r8g.16xlarge and they are actually r8gd.16xlarge (notice the d) which comes with directly attached nvme disks.

▲

svuiv 8 hours ago | parent | next [-]

We used r8g.16xlarge instances with EBS disks, no nvmes

▲

whalesalad 7 hours ago | parent [-]

would love to hear more about the ebs volumes, iops/size/raid configuration

	▲	svuiv 7 hours ago \| parent [-]
		each shard had a 4TiB volume with 65k IOPS and 1,500 mbps of throughput

▲

rcrowley 7 hours ago | parent | prev [-]

It felt rude to take so many r8gd instances away from our customers who really love those (and i8g and i8ge).