I'm sad that drives don't have a 'shutdown' command which writes a few extra bytes of ECC data per page into otherwise empty flash cells.

It turns out that a few extra bytes can turn a 1 year endurance into a 100 year endurance.

Because no one is willing to pay for SLC.

Those QLC NAND chips? Pretty much all of them have an "SLC mode", which treats each cell as 1 bit, and increases both write speeds and reliability massively. But who wants to have 4 times less capacity for the same price?

Endurance going down is hardly a surprise given that the feature size has gone down too. The same goes for logic and DRAM memory.

I suspect that 2035 years time, hardware from 2010 will work, while that from 2020 will be less reliable.

I also seem to remember reading retention is proportional to temperature at time of write. Ie, best case scenario = write data when drive is hot, and store in freezer. Would be happy if someone can confirm or deny this.

On the other hand when capacity goes up, the cycle-count goes down for the same workload. A 4TB drive after 1K cycles has written the same amount of data as 100GB drive after 40K cycles.

That's how it has to work. To increase capacity you have to make smaller cells where charge may easier diffuse from one cell to another. Also to make drive faster, stored charge has to be smaller, which also decrease endurance. With SLC and QLC comparison is even worse as QLC is basically clever hack to store 4 times more data in the same number physical cells - it's tradeoff.

So AWS S3 Glacier might actually be cold

> Even a QLC SSD that has only been written to once, and kept in a freezer at -40, may hold data for several decades.

So literally put your data in cold storage.

SSD firmware engineer here. I work on enterprise stuff, so ymmv on consumer grade internals.

Generally, the data refresh will all happen in the background when the system is powered (depending on the power state). Performance is probably throttled during those operations, so you just see a slightly slower copy while this is happening behind the scenes.

The unused space decaying is probably not an issue, since the internal filesystem data is typically stored on a more robust area of media (an SLC location) which is less susceptible to data loss over time.

As far as how a user is supposed to manage it, maybe do an fsck every month or something? Using an SSD like that is probably ok most of the time, but might not be super great as a cold storage backup.

Keep in mind that when flash memory is read, you don't get back 0 or 1. You get back (roughly) a floating point value -- so you might get back 0.1, or 0.8. There's extensive code in SSD controllers to reassemble/error correct/compensate for that, and LDPC-ish encoding schemes.

Modern controllers have a good idea how healthy the flash is. They will move data around to compensate for weakness. They're doing far more to detect and correct errors than a file system ever will, at least at the single-device level.

It's hard to get away from the basic question, though -- when is the data going to go "poof!" and disappear?

That is when your restore system will be tested.

Typically unused empty space is a good thing, as it will allow drives to run in MLC or SLC mode instead of their native QLC. (At least, this seems to be the obvious implication from performance testing, given the better performance of SLC/MLC compared to QLC.) And the data remanence of SLC/MLC can be expected to be significantly better than QLC.

> I assume this blog is a re-hash of the JDEC retention standards[1].

Specifically in JEDEC JESD218. (Write endurance in JESD219.)

In the longer JEDEC overview document[1] it explains that in the ideal 'direct' testing method retention testing is only performed after the endurance testing. Which is only after the drive has had its max spec'd TBW written to it.

While if the endurance testing would exceed 1000 hours an extrapolated approach can be used to stress below the TBW but using accelerated techniques (including capping the max writable blocks to increase wear on the same areas).

Which is less dramatic than the retention values seem at first and than what gets communicated in articles I've seen. Even in the OP's linked article it takes a comment to also highlight this, while the article itself only cites its own articles that contain no outside links or citations.

[1] https://www.jedec.org/sites/default/files/Alvin_Cox%20%5BCom...

With 1 year power-off retention you still loose data, so still a compromise on data retention

What does power-cycle mean in this case?

I've seen this topic come up a lot on a datahoarder forum but there was never any consensus on what it would take to assure the data was refreshed because no one really knew how it worked. People would assume anything from a short power up to simply rewriting all data as possibilities.

The firmware engineer responding up thread is actually the best information I've seen on this. And it kind of confirmed my suspicions. He works on enterprise drives and they run a refresh cycle as background cycle that seems to start on a random timer. I didn't see any response (yet) on how long this process takes but I think we can infer that it takes a while based on it being a background process. Probably hours or more. And this is all for an enterprise drive, I wouldn't be surprised if consumer drives have a less robust or even non-existent refresh function.

I'm of the opinion that that based on the little information about some implementations of this function, the logical conclusion is you should just rewrite all of the data over again on your cold backups for the cycle. Powering it on isn't enough, even powering it on for a day might not do anything. As you said this is a pretty big drawback for uses flash as a cold backup.

I think we should stop considering flash as permanent storage. It is temporary storage that keeps working as long as given power.

I wish there was archival storage something long-lasting and large capacity. That would make a good complement to flash for writing backups and long-lasting data.

What is the goldielocks area here, spinning rust? My 20 year old ide hdds seem to be… ok

I was talking to a lawyer recently and somehow this subject came up and he confessed he has been scanning and storing all his files onto SSDs for years which he keeps in a safe to meet a 7-year retention requirement.

If full rewrite helps doesn't it sound like TRIM implementation in SSDs is buggy or insufficient? Or internal cell wear-maps aren't detailed enough. Anyway plenty ways it can go wrong, SSD firmware had also plenty of high profile bugs, including total bricking.

There are lots of other potential causes for the same effect...

Eg. Data structures used for page mapping getting fragmented and therefore to access a single page written a long time ago requires checking hundreds of versions of mapping tables.

no, the firmware does any maintenance. good firmware should do gradual scrub whenever it's idle. unfortunately, there's no real way to know whether the firmware is good, or doing anything.

I wonder if there's some easy way to measure power consumed by a device - to detect whether it's doing housekeeping.

Maybe tangentially related to my Pixel phone losing photos [0]?

[0] https://news.ycombinator.com/item?id=46033131

I also need an answer to this.

You should absolutely be doing a full block read of your disk, dd if=/dev/disk of=/dev/null every couple weeks

We really don't know. One thing I wish some of these sites would do is actually test how long it takes for the drives to decay and also do a retest after they have been left powered for say 10 minutes to an hour, read completely, written to a bit etc and see if they can determine what a likely requirement is.

The problem is the test will take years, be out of date by the time its released and new controllers will be out with potentially different needs/algorithms.

I'd imagine full read of the whole device might trigger any self-preservation, but I'd also imagine it's heavily dependent on manufacturer and firmware

I think that reading all of the information from the SSD should “recharge” it in most cases. The SSD controller should detect any bit flips and be able to correct them.

However, this is implementation detail in the SSD FW. For Linux UBI devices, this will suffice.

I too wonder about this. I'd love to see someone build a simulated "fast decay" SSD which can show how various firmware actually behaves.

Back it up. 1 2 3.

Read off all live data on the drive. This should cause the nand management firmware to detect degrading cells via ECC and move the data in order to refresh isolated cell voltage levels.

I would run something like CHKDSK, or write a script to calculate a hash of every file on disk.

No idea if that's enough, but it seems like a reasonable place to start.

If you are using a backup program like "kopia" there are special commands to recheck all hash blocks.

You just can't trust the hardware to know how to do this, need backup software with multiple backup locations, it will know how to recheck integrity

The simplest trick is just don't use SSD for long term backup, use a normal magnetic hard drive instead, those thing way lasts _longer_ (but not forever, even in human timescale).

I have a HDD that was 17+ years since it last powered on. I dug it out recently to re-establish some memories, and discovered that it still reads. But of course you need to take care of them well, put them in an Anti-Static Bag or something similar, and make sure the storage environment is dry.

It's not prefect, but at least you don't have to struggle that much maintaining SSDs.

There are cheap USB<->PATA/SATA or USB<->NVME adapters out there that usually also come with 120v AC -> 3/5/12v DC PATA/SATA power supplies (and if the SATA SSDs only need 5v then some adapters might work with USB alone).

I use them for working with old unmounted hard drives or for cloning drives for family members before swapping them. But they would probably work for just supplying power too?

The one I use the most is an 18 year old Rosewill RCW-608.

I don't know if the firmware/controller would do what it needs to do with only power connected. I wonder if there's some way to use SMART value tracking to tell? Like if power on hours increments surely the controller was doing the things it needs to do?

Yes, NOR Flash guarantees 20 years data retention. It's about $30/GiB.

There are also expensive high-grade SLC NAND flash chips available that offer significantly higher retention time than the cheaper commodity channel TLC NAND (i.e. 10 years versus 3 years).

In general, though, whether NAND or NOR, the fundamental way that flash works is by creating an isolated voltage charge for each bit (or several bits, for TLC), making it effectively a vast grid of very tiny batteries.

Like all batteries, no matter how well-stored, they will eventually leak energy to the point where the voltage levels change enough to matter.

Further, it's not enough to simply make power available to this grid since the refresh of the cells requires active management by a nand controller chip and its associated software stack.

here's a better idea, buy a mechanical hard disk.

How much are you saving vs an always-on ARM board with M.2 slots? Is it worth the pennies?

> Could be battery-powered.

How often does it need to run? If it could be solar powered you could probably avoid a whole bunch of complexity per unit longevity.

Isn't that the SSD controller's job?

Huh. I wonder if this is why I'd sometimes get random corruption on my laptop's SSD. I'd reboot after a while and fsck would find issues in random files I haven't touched in a long time.

All the big 3D NAND makers have already switched from floating gate to charge trapping. Basically the same as what you describe but basically the electrons get stuck in a non-conductive region instead of on an insulated gate.

> I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.

You can run an error-correcting code on top of the regular blocks of memory, storing, for example (really an example; I don’t know how large the ‘blocks’ that you can erase are in flash memory), 4096 bits in every 8192 bits of memory, and recovering those 4096 bits from each block of 8192 bits that you read in the disk driver. I think that would be better than a simple “map low levels to 0, high levels to 1” scheme.

I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.

There is a way to turn QLC into SLC: https://news.ycombinator.com/item?id=40405578

> I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.

Loads of drives do this(or SLC) internally. Though it would be handy if a physical format could change the provisioning at the kernel accessible layer.

> Honestly I can’t quite believe mlc works at all, let alone qlc. I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.

Manufacturers often do sell such pMLC or pSLC (p = pseudo) cells as "high endurance" flash.

the market demands mostly higher capacity

tlc/qlc works just fine, it's really difficult to consume the erase cycles unless you really are writing 24/7 to the disk at hundred of megabytes a second

At the physical device layer (i.e. what a nand controller vendor programs to), a nand flash device that supports ONFI 2 SLC Mode can be configured to use (some of) its blocks in SLC mode rather than MLC/TLC/QLC/etc. This allows one to divide the array into high-reliability versus high-capacity regions.

The self healing only works if you have redundant drives. If all SSD’s are like this maybe they need alarms that sound as an onboard battery gets low to ensure someone plugs them in.

I think you're talking about two different things; "adaptives" are usually stored in EEPROM or the MCU's NOR flash, which is basically better-than-SLC levels of reliability, especially as they're written once and treated as ROM after that.

OptiNAND is a "SSHD" and thus has the same concerns with retention as an SSD. https://en.wikipedia.org/wiki/Hybrid_drive

I wonder too. I don't trust SSDs/ flash for my archives, hence I'm stuck on max. 18TB drives atm.

shouldn't really be a problem, since the capacity required for this use is so low.

the real issue here is QLC in which the flash cell's margins are being squeezed enthusiastically...

Yeah. That part was so out of touch, I have to wonder who they're carrying water for. Most people backup their data? What planet does the author live on?

Interesting statement. I suppose you could argue that long term archival storage is communication with a long tailed, sometimes ill-defined window for receiving the message. Few people write books they don't want anyone to read.

Yes, it applies to all flash memory. Timescales will vary depending on the fabrication type, ambient temperature etc etc.

AIUI, informal tests have demonstrated quite a bit of data corruption in Flash drives that are literally so worn out that they might as well be about to fail altogether - well beyond any manufacturer's actual TBW specs - but not otherwise, least of all in new drives that are only written once over for the test. It seems that if you don't wear out your drive all that much you'll have far less to worry about.

this is about drives that are not plugged in. are you saying parity would let you simply detect that the data had gone bad? increasing the number of drives would increase the decay rate, more possibilities for a first one to expire. if your parity drive expired first, you would think you had errors when you didn't yet.

How would this work? Wouldn't all these drives start loosing data at roughly at the same time?

I learned this when both my old laptops would no longer boot after extended off power time (couple years). They were both stored in a working state and later both had SSDs that were totally dead.

I could be wrong, but I believe the general consensus is along the lines of "SSDs for in-use data, it's quicker and wants to be powered on often. HDDs for long-term storage, as they don't degrade when not in use nearly as fast as SSDs do.

or you could power them on 1-2x /year.

Usually described as bit rot.

USB thumb drives (and SD cards) are bottom-of-the-barrel flash, so yes, absolutely.

That's kicking the can down the road for double the cost. Only a backup on spinning rust is actually a backup.

Furthermore, replication isn't a backup.

all the major players say "tape". (but that's partly for practical issues like scaling and history)

Depending on your time requirements, carving it into a stone tablet is generally a good choice

Who do you pay for this? (To rephrase : which cloud storage vendors do you use?) interested in the $2/month price point :)

Do we actually know the clouds will do this? S3 is just about coming to its 20th anniversary.

Long enough to experience data rot to a small degree but realistically what proportion of users have archived things away for 10+ years then audited the fidelity of their data on retrieval after fetching it from Glacier

Any given TBW/DWPD values are irrelevant for unpowered data retention. Afaik, nobody gives these values in their datasheet and I'm wondering where their numbers are from, because I've never seen anything official. At this point I'd need to be convinced that the manufacturers even know themselves internally, because it's never been mentioned by them and it seems to be outside the intended use cases for SSDs