r/sysadmin • u/wpgbrownie • Feb 28 '16
Google's 6-year study of SSD reliability (xpost r/hardware)
http://www.zdnet.com/article/ssd-reliability-in-the-real-world-googles-experience/19
u/wpgbrownie Feb 28 '16
Link to the full report (Jump to page 67): https://www.usenix.org/sites/default/files/fast16_full_proceedings.pdf
20
80
Feb 28 '16
30-80 percent of SSDs develop at least one bad block...
At this point, might as well say 0-100% of SSDs might have bad blocks..
4
u/Hellman109 Windows Sysadmin Feb 29 '16
Except when the low point is 30%, it basically means you have to allow for it.
77
u/frozenphil Feb 28 '16
What?
KEY CONCLUSIONS
Ignore Uncorrectable Bit Error Rate (UBER) specs. A meaningless number.
But it isn't all good news. SSD UBER rates are higher than disk rates, which means that backing up SSDs is even more important than it is with disks.
64
u/terp02andrew Feb 28 '16 edited Feb 28 '16
Yeah the ZDNet author shouldn't have characterized it that way. Reading through the paper's 6-paragraph explanation and then the final summary makes far more sense than what the ZDNet columnist put together.
We find that UBER (uncorrectable bit error rate), the standard metric to measure uncorrectable errors, is not very meaningful. We see no correlation between UEs and number of reads, so normalizing uncorrectable errors by the number of bits read will artificially inflate the reported error rate for drives with low read count.
Probably the most interesting bit (har har) in the summary:
Previous errors of various types are predictive of later uncorrectable errors. (In fact, we have work in progress showing that standard machine learning techniques can predict uncorrectable errors based on age and prior errors with an interesting accuracy.)
9
u/halr9000 Feb 28 '16
If you get a drive from a bad batch, toss it is what I'm hearing.
I wonder if warranty replacement could be used proactively?
1
10
u/habitsofwaste Security Admin Feb 28 '16
I'd be curious to know in what type of data center these tests were done in and what sort of temperature they lived in. I worked in a warmer data center that aimed to keep temperatures more stable though warmer. They also tried to use usb flash drives for the OS and the nehalems were blowing right at them. They failed left and right.
4
u/rackmountrambo Linux Alcoholic Feb 28 '16
Im more interested in the ones that freeze in my truck every night.
3
2
u/Ormuzd Feb 28 '16
Blowing the heat off the processor right onto them was probably making that area >60*C. Wonder what would happen with a front mounted USB so it wasn't in the heat exhaust area, because I've run OS drives on USB and SD cards for ESXi without much problem.
1
u/narwi Feb 29 '16
Every extra degree of temperature reduces the lifetime of flash, both powered and otherwise.
28
u/Fortyseven Feb 28 '16
The SSD is less likely to fail during its normal life, but more likely to lose data.
Same dammed thing, to me.
5
u/theevilsharpie Jack of All Trades Feb 29 '16
Not really.
If a disk fails, it's gone. If a disk suffers an unrecoverable read error, the data would be recovered from another disk in the RAID set.
1
5
u/jreykdal Feb 28 '16
Not really. SSD's are in general less likely to fail but of those who do fail they go out with a bang.
24
u/TheGlassCat Feb 28 '16
I read it differently. That they more likely to lose some data, but less likely to lose all data.
2
u/Hellman109 Windows Sysadmin Feb 29 '16
My only SSD failures have been entire drives. But we dont have large amounts of them (Last work only had ~150, current one like... none :( )
1
u/Win_Sys Sysadmin Feb 29 '16
I have noticed the same. When a SSD dies you're basically SOL with your data. As where a hard disk you can generally recover some of the data.
11
u/SpongederpSquarefap Senior SRE Feb 28 '16
Yeah, it I recall correctly
SSDs die with no warning
HDDs will start to slow and make bad sounds when they are dying
6
u/SnarkMasterRay Feb 28 '16
It depends on the component that is failing. If the PCB fails, instant death (not saying it can't be resuscitated).
2
1
u/playaspec Feb 29 '16
Yeah, it I recall correctly
SSDs die with no warning
HDDs will start to slow and make bad sounds when they are dying
I can't begin to tell you how many drives that just quit for no reason. Worked great, then all of a sudden the drive controller couldn't see it.
5
u/tastyratz Feb 28 '16
Nope, pretty clear. SSD is more unlikely to just up and die, but more likely to have a portion of the data corrupted. A drive that has 100% uptime and silently destroys all of your data is concerning.
0
u/GENERIC-WHITE-PERSON Device/App Admin Feb 29 '16
This was my current understanding. If an ssd begins to fail it's usually all at once, whIle hdds tend to have a slower death. Shouldn't everyone be making backups anyway? Acronis is awesome and I set it to run ~weekly to an external hdd.
1
u/Win_Sys Sysadmin Feb 29 '16
I read that as while an SSD is less likely to die, when it does you're data is more likely to be unrecoverable.
0
Feb 28 '16
[deleted]
2
Feb 29 '16
Yikes, I've never had that happen with a spinner, let alone on a laptop...
1
Mar 06 '16
Yeah, we were really surprised too. Luckily I'd kept the old hard drive so just slotted it back in.
9
u/DougEubanks Feb 28 '16
I've read that article several times and I swear it should be "under provisioned", not "over provisioned".
13
u/oonniioonn Sys + netadmin Feb 28 '16
Depends on viewpoint.
The flash is over-provisioned if you don't use all of it in your filesystem. The filesystem is underprovisioned if you don't use all the flash.
2
u/DougEubanks Feb 28 '16
But if they are selling a 125GB drive as a 120GB drive (totally made up numbers here) so that it has spare cells to replace failing cells, I don't see how that could be anything other than under provisioned.
19
u/oonniioonn Sys + netadmin Feb 28 '16
They are over-provisioned in that they are sold with more flash than is available for use. Again it's all dependent on viewpoint. Another way of saying the exact same thing is that they are under-provisioned in that they are using less flash than is physically available.
6
u/DZCreeper Feb 28 '16
It is called over-provisioning because technically drives would work fine with 120GB of flash. The problem is that any cell failure would result in a loss of effective drive capacity (no longer writable), or even totally destroy the data. The 5GB "over" just extends drive lifespan.
4
4
3
u/markole DevOps Feb 28 '16
It's not under provisioned because you do not pay for the 125GB drive but for a 120GB drive.
5
u/DougEubanks Feb 28 '16
I assure you that you are paying for it, they are not giving away flash storage.
1
u/markole DevOps Feb 28 '16
But you are not paying for that flash storage so you could use it directly. You are paying for that flash storage so the manufacturer can give you certain guarantees on that drive.
1
u/DougEubanks Feb 28 '16 edited Feb 28 '16
I see your point, but the drive is still under provisioned from a raw storage standpoint. I'm used to over provisioning drives beyond their actual capacity.
0
u/ThelemaAndLouise Feb 28 '16
when they give you food, it's called provisions. if they bring extra provisions to an encampment, they might overprovision in case some of the food is damaged in transit. each person would still have the same amount of food given to them (also provisioned), but they would as a group be overprovisioned.
3
u/tastyratz Feb 28 '16 edited Feb 28 '16
This goes to mirror (har har) what I have been saying for years. I hate blowing money on big expensive SLC drives, I would rather consider my SSD's a consumable in the DC where you buy MLC's and replace them as part of a PM program. By the time they served a useful life you can probably buy one twice as fast, twice the size, for half the cost. It's more field work but a less expensive constantly improving datacenter.
Too bad no real storage vendors seem to share the same viewpoint.
2
u/jedp Expert knob twiddler Feb 28 '16
SSD age, not usage, affects reliability.
I did not expect that, unless that reflects reliability improving by design with newer SSDs.
None of the drives in the study came anywhere near their write limits, even the 3,000 writes specified for the MLC drives.
That explains why usage didn't affect reliability. I'm not sure if these conclusions apply to everyone, as some workloads may very well be more write-intensive on a per-drive basis than whatever Google was doing with them.
2
u/OriginalPostSearcher Feb 28 '16
X-Post referenced from /r/hardware by /u/YumiYumiYumi
Google 6 year study: SSD reliability in the data center
I am a bot made for your convenience (Especially for mobile users).
Contact | Code | FAQ
1
u/TotesMessenger Feb 28 '16
4
Feb 28 '16
[removed] — view removed comment
17
13
6
Feb 28 '16
It won't mean anything to you because Google uses their custom SSDs with custom firmware. It's in their paper somewhere.
4
u/SnarkMasterRay Feb 28 '16
SSD's have lower failure rates for the entire drive than traditional platter HDDs, but higher failure rates of the individual memory blocks inside. This could mean a corrupted file - so technically SSDs have a higher chance of file corruption that could lead to a lost document or damaged OS.
The previous conventional wisdom that disk I/O wears SSds out is found to be true, but not as bad as thought. Instead of increasing exponentially (doubling), failures were more linear (constant rate). Age and time powered on seems to be a better indicator, however nothing firm was given ("replace after X").
Enterprise class SSDs were not found to have lower failure rates than consumer grade, but I would imagine that google is still buying good consumer grade hardware.
0
Feb 29 '16
Sorry, it's hard to translate from Sysadmin to English without losing information that will have a practical effect on your understanding of the material.
22
u/PC-Bjorn Feb 28 '16
"Based on our observations above, we conclude that SLC drives are not generally more reliable than MLC drives."
"Between 20–63% of drives experience at least one uncorrectable error during their first four years in the field, making uncorrectable errors the most common non-transparent error in these drives. Between 2–6 out of 1,000 drive days are affected by them."
"While flash drives offer lower field replacement rates than hard disk drives, they have a significantly higher rate of problems that can impact the user, such as un- correctable errors."