The problem with heavy swapping on NVMe (or other flash memory) is that it wears out the flash storage very quickly, even for seemingly "reasonable" workloads. In a way, the high performance of NVMe can work against you. Definitely something you want to check out via SMART or similar wearout stats.

▲ ciupicri 3 days ago | parent | next [-]

For what it's worth, these are the lifetime estimates for the Micron 7450 SSD [1]:

    Model  Capacity  4K Rand  128K Seq
               [GB]    [TBW]     [TBW]
    PRO        3840    7_300    24_400
    PRO        7680   14_000    48_800
    MAX        3200   17_500    30_900
    MAX        6400   35_000    61_800

> Values represent the theoretical maximum endurance for the given transfer size and type. Actual lifetime will vary by workload …

> Total bytes written calculated assuming drive is 100% full (user capacity) with workload of 100% random aligned 4KB writes.

[1]: page 6/17, https://assets.micron.com/adobe/assets/urn:aaid:aem:d133a40b...

▲ inkyoto 3 days ago | parent | prev | next [-]

Not an issue for the commenter – since they have mentioned S3, they are either using AWS EBS or instance attached scratch NVMe's which the vendor (AWS) takes care of.

The AWS control plane will detect an ailing SSD backing up the EBS and will proactively evacuate the data before the physical storage goes pear shaped.

If it is an EC2 instance with an instance attached NVMe, the control plane will issue an alert that can be automatically acted upon, and the instance can be bounced with a new EC2 instance allocated from a pool of the same instance type and get a new NVMe. Provided, of course, the design and implementation of the running system are stateless and can rebuild the working set upon a restart.

▲

jitl 3 days ago | parent [-]

EBS is slow. No way we would use it for swap. Gotta be instance storage device. And yes, we can rebuild a node from source data, we do so regularly to release changes anyways.

	▲	inkyoto 3 days ago \| parent [-]
		I figured that you were using instance attached NVMe's since you mentioned the scale of your load – an EBS even with the io2 Express storage class can't keep up with a physical NVMe drive on high intensity I/O tasks. Regardless, AWS takes care the hardware cycling / migration in either case.

▲ jitl 3 days ago | parent | prev | next [-]

Let’s say we’re spending $1 million on hardware hypothetically with the swap setup.

At that price point, either we use swap and let the kernel engineers move data from RAM to disk and back, or we disable swap and need user space code to move the same data to disk and back. We’d need to price out writing & maintaining the user space implementation (mmap perhaps?) for it to be fair price comparison.

To avoid SSD wear and tear, we could spend $29 million a year more to put the data in RAM only. Not worth!

(We rent EC2 instances from AWS, so SSD wear is baked into the pricing)

▲ p_ing 3 days ago | parent | prev | next [-]

While what you stated is overall not true, who cares with a 97% cost savings vs RAM? Just pop in another NVMe when one fails.

▲ justsomehnguy 2 days ago | parent | prev | next [-]

> that it wears out the flash storage very quickly

Only if you use a consumer grade flash with a non-consumer grade usage.

For anything with DPWD >= 1 it's not an issue, eg:

https://news.ycombinator.com/item?id=45273937

▲ man8alexd 3 days ago | parent | prev [-]

[dead]