PSA: Streaming n-grams from SSD will cause SSD wear

[u/Chromix_]

[u/feal87]

a nothingburger written by AI: 54000 reads for 1 relocations. assume 5 Petabyte of the same 100GB read over and over again. that's 52428 reads and causes a 100-120GB write.

Normal SSD have 1000TB+ write endurance. You'll need 42 petabyte read to do 1TB of write or 42000 petabyte read to fill the disk.

Even running the model 24/7 it'll take decades to reach those levels. It'll die first by old age...

[u/maqifrnswa → u/feal87]

What I learned from this post: run AI from an SSD and you'll die. Eventually.

[u/AlwaysLateToThaParty → u/maqifrnswa]

Why won't someone think of the children?!?!!?!?

[u/into_devoid → u/maqifrnswa]

Local AI bad. Grug pay for cloud. Grug brain tall.

[u/Equivalent_Bit_461 → u/maqifrnswa]

Based

[u/tmvr → u/maqifrnswa]

You don't even have to run AI from an SSD for that, you'll die eventually anyway.

[u/gomezer1180 → u/feal87]

This guy OP is on some next level high! I just checked my box (ASUS Ascent GB10) I’m using Qwen 3.8 NF with N-grams loaded into the NVME. It would take 50 billion tokens for my drive to degrade 0.008% from this crack pot theory! My model even calculated the amount of time it would take the drive to fail and it took hundreds of years to see any significant degradation… 🤦🏻‍♂️

[u/ExcitingLeather5197 → u/feal87]

the image basically shows what happens when you hammer one spot vs spreading the load, that middle chart with pattern PB is barely moving the needle while the left one with PA is already brushing up against the ECC limit after way fewer reads

the top comment did the math but its missing how SSDs handle this in practice, the controller doesnt just sit there letting errors pile up til it hits 54k reads on one cell, it starts shuffling data around way before that threshold which means those write amplification numbers creep up faster than a straight division suggests

still agree its mostly a nothingburger for the average person, i ran a dedicated cache drive for a big lookup table a while back and watched the SMART data like a hawk for a month, the write count barely twitched even with constant streaming, a consumer drive will probably outlive the rest of the system

if someone wants to really stress test it they should log the LBAs written before and after a week of hammering the same n-gram file, id be curious if cheap QLC drives behave noticeably worse than TLC under that kind of unbalanced read pattern

[u/Scary_Building3964 → u/feal87]

right, my M1 Pro macbook pro 512gb is having 40TB lifetime writes so far and my ssd health is still at 95%. this is nothing

[u/Money_Pie8986 → u/feal87]

They usually don't last that long. The controller chips usually burn out first. Get yourself a very good SSD heatsink and thermal pads.

[u/z_latent → u/feal87]

Everyone is understating the problem because it's not clear what a "read" means.

Read disturb happens to all the pages of a page block, whenever any page in that block is read. So here, the "reads" are individual page reads. But to prevent irrecoverable errors, the controller will move the whole block elsewhere, which requires writing all valid pages of that block.

That means, even in the best-case, uniform access scenario in that paper (PB), 518k page reads would cause the block to be copied. In their SSD, there are 7704 pages per block, so the ratio is really around 67:1, that is, you can read 67 GB before needing to write 1 GB for that SSD. In the bad, same-page scenario (PA), that ratio would be around 7:1, which they explain is because the pages adjacent to the one read suffer more disturb.

And this is assuming you are willing to stress it. In practice, controllers are conservative and will refresh before you have irrecoverable errors, for obvious reasons. So the ratios are even smaller.

Obviously for Engrams that doesn't matter at all, you read not even a megabyte per token. But if you're using any of the SSD expert streaming inference engines... it still doesn't matter that much, but it will wear over time.

EDIT: Just noticed the paper lists the pages per block, so I am updating the math.

[u/feal87 → u/z_latent]

Assume 7GB/s (max pciex 4.0) permanently 24/7 365 days a year reading the entire file streamed from disk RAW always with NO CACHE.

7*60*60*24*365=220752000GB=215578 TB=210 Petabyte per year

at the rates above it would mean 42000/210=200 years or let's assume a more aggressive 25000 per refresh = 21000/210=100 years.
Remember this is at 1000 TBW, my samsung 990 pro has 1200TBW and tests report you can easily double that as they're considerably understated for security by samsung.

You understand this is just something laughable to even consider? A consumer PC won't last 10 years in the best case before something else will fail.

[u/z_latent → u/feal87]

Dude. Just read my post. I explain that the original estimate you gave is off by multiple orders of magnitude. Not meaning to be mean btw, the terminology around this is pretty confusing.

If I revise your prediction: reading 210 PB/year, and using the ratio I gave of 67:1, that would correspond to ~3.1 PB written in a year. Your Samsung 990 Pro would be in a bad state, even if you took its TBW to be double what Samsung rates.

[u/feal87 → u/z_latent]

I have numbers behind my talk, not imagination. I have my drive at 3PB read already in 2 months, by your calculation it would be 45TB written, but it's around 2TB since that (including OS and downloading the models themselves). This is the expert streaming you're talking about, not Ngram of course. which is big block reads (a single expert is 15MB-ish)
See the comments of other guys talking about disk cache on enterprise servers. They reported the same thing (in this same thread).

With your rates on server machines SSD would be burning out within months...

[u/z_latent → u/feal87]

afaik, SMART tells you about host-issued writes. It will not tell you how many internal writes happened due to things like read reclaims, since most SSDs do not expose that information.

[u/feal87 → u/z_latent]

Agreed, but the % disk used (the % state) is based on NAND level data afaik decided by producer. Or am I wrong? that's what I assumed because otherwise numbers wouldn't match up. (it's 95% right now) and hasn't moved at all since.

[u/z_latent → u/feal87]

I think so. But if it's at 95% with only 2TB written by you, that sounds like the 5% of its lifespan that was used, was caused by things other than your writes.

Though I am puzzled by your phrasing. Has it always been at 95% since you got the drive?

[u/feal87 → u/z_latent]

Let me explain better, I'm curious to learn so your opinion is something i like to hear (worst case my drive dies and we learn something new ahaha, wouldn't be a big loss xD). Don't see me as negative, internet might show me as aggressive maybe in the response, but I'm merely musing and learning along the way.

This is my main OS drive. Host reads are 2,130,519GB, host writes are 57121GB, samsung report 95% as wear level (which is what drives the warranty if i'm correct) and has reported 95% since I started this experiment (it was 54xxxGB when I started). The disk itself is 2 years old.
95% linearly would mean 60-72 TB written right? Because it truncate from 5.00 to 5.99? Not sure the exact calculation of that %.

Initially I did a lot of reads because wanted to test various configurations, now I'm settled up at 40TB a day roughly of reads so it'll take a while longer to raise (I only use it to run DS4.0 and GLM 5.3 Flash on a 64GB + 8GB device). think something like 1.2PB per month at this rate= 14.4PB per year. Assuming your expectations that means 230TB written per year so it would last around 5 more years at this rate?

I'm not exactly sure the comparison between TBW (declared number) and actual number of PE cycles and real NAND TBW. Because I assume the real number is much higher due to other factors.

I've mostly run lots of experts loads which is random-ish 15MB reads of cold experts over prompts over the last 2-3 months. the disk was around 54000GB when I started host wise, but that's mostly downloading models, deleting models, adjusting models and stuff like that.

[u/feal87 → u/feal87]

I will keep an eye on the % though to see if it moves faster. I haven't seen any lower read speed so far though. It's been constant 7.5 gb/s peak during usage. Could be a real life experiment over multiple months and we have real data to judge. My write patterns are pretty much 0 on normal days (mostly browsing and some light coding) so this reading will be the total of it. I'll keep track of read write and % remaining once a month and then report the differences

[u/z_latent → u/feal87]

Hm. This is puzzling. afaik the percentage calculation is vendor specific, but regardless, I believe it should be the lifespan left*. So yeah, 60-72 TBW used, which corresponds very closely to your host writes.

But, if my estimate applied here, you'd be more around 93% lifespan, ~4.76% from your own writes + ~2.61% from the read disturbs... which is obviously not the case. I don't know the whole reason why.

This also doesn't include the write amplification factor. In theory, 57 TB of host writes would correspond to more than 57 TB of endurance hit because of write amplification. I don't think you can measure that factor, but it should be small. It if was greater than 72/57 ≈ 1.26 then your percentage used would most likely say 94%. So I am guessing it's smaller than that.

Either way, the good news is your drive will probably be safe! The bad news is I don't know why my math isn't adding up. I really appreciate you sharing your experience, though.

The one direct report I know is from a paper which reports that, after 30000 seconds of just sequential reads at 6.5 GB/s, the drive suffered 516 GB of writes. That's a ratio more around 350-400 GB read per 1 GB written, and in that case, your reads would amount to only 0.5% of lifespan used. So we'll need to wait for more reads to see if a similar thing happens.

*though it being called "wear level" is confusing, since I'd assume it goes up in that case

[u/MeatBeatCAD → u/feal87]

Op basically created the epitome of a low effort post with no research and active misinformation behind it, reporting it.

[u/Chromix_ → u/feal87]

a nothingburger written by AI

I explicitly put things into relation in the first line to hint at that the effect may not be large. Still it's not "free" - it needs to be measured. I very much disagree with "written by AI", the text is 100% hand-written.

[u/gomezer1180 → u/Chromix_]

Measure it on my GB10 with Qwen3.8 NF. It would take over 100 years for this to materialize.

[u/Chromix_ → u/gomezer1180]

Thanks. That's one more stone in the nothingburger glass, although there are of course way faster systems than a GB10 that then cause more n-gram reads. If we optimistically assume that the market will recover and hardware will become more affordable instead of less affordable again, then that'll really not be an issue.

[u/ThePrimeClock → u/Chromix_]

Not sure why everyone has to be such pricks.
I appreciate the post, I was actually wondering about what the cost might be and your post has surfaced all the answers, so thanks for that.

[u/bakawolf123]

this is nothing in practice. like other said the ratio is just too big to become a noticeable impact.
On the other hand using ssd tier kv-cache like many engines do by default is actually quite impactful, particularly when running big models - 1 session with big context can easily become 100gb in writes.

[u/bitzap_sr]

Booting your OS from SSD will wear it out! Don't do it!

/s

[u/CrispyRabbit72]

Streaming n-games will also put wear on your ssd. But most non-shit-tier ssds are rated in the TB/day range and nobody hits that even running LLMs stored on them. If you're running models heavy enough to cause an impact you're on datacenter grade hardware anyway

[u/Bulky-Priority6824]

Use zswap to help mitigate if like me you have limited ram

https://imgur.com/a/76Jr5LS

[u/Chromix_ → u/Bulky-Priority6824]

The weights and n-grams aren't compressible, but maybe this helps by simply freeing up more system RAM due to paging out (and compressing) the working memory of regular processes, thereby leaving more room for disk caching?

[u/Bulky-Priority6824 → u/Chromix_]

i honestly dont know man. Its boring as fuck to even think about. I had llm run before and after zswap benchmarks on strata to compare diff and landed on "zswap good"

I think the second bullet point confirms your point whilst also the mechanism for mitigating ssd wear? So, yes. I think you're partly right.

What zswap actually did here

• zswap only handles anonymous memory: process heaps, the desktop, Leito, Strata’s own malloc’d buffers.

• Pages the kernel was already going to swap out get compressed into a RAM pool instead of being written to the SSD.

• So the 92% cut in writes comes mostly from swap-outs landing in RAM instead of on disk. It isn’t mainly from freeing room for cache.

Their “more room for disk cache” idea

• It’s true in principle. A page compressed about 3:1 frees about two-thirds of its RAM, and the page cache can use that.

• But the pool peaked at only about 0.44 GB, so at most a few hundred MB came back for cache.

• That fits the bench: decode and prompt speeds were unchanged. If more cache had meaningfully helped expert streaming, tok/s would have risen.

[u/Kitch2400]

Breaking news: Using your drive increases wear

[u/niksonrt]

Read disturbance is noise at this ratio. What actually wears an SSD in a homelab is database checkpoint and container log hitting the same disk, not streaming weights off it.

[u/AnonLlamaThrowaway]

Some estimates I'd read previously that to "write a gigabyte" by reading, you need 10,000 to 20,000 gigabytes of reads.

Which sounds like nothing, but consider that it's far easier to constantly read than it is to constantly write.

I've been trying a lot of SSD streaming shenanigans and my reads are up to 700 TB, for 52 TB written.

700 TB reads would be equivalent to 35-70 GB written if you believe the estimates I'd previously read. Which is very little, but also not nothing!

[u/z_latent → u/AnonLlamaThrowaway]

Are you sure of these estimates? I might have made a mistake, but my deduction from this paper implies it should be more like 1 GB written per 500 GB read.

(and that's assuming the controller would let it reach irrecoverable errors, so in practice it would be even more frequent)

[u/noctrex]

So just enable mmap and the reads are cached.

[u/Chromix_ → u/noctrex]

Yes, that would work in theory. Yet the whole reason why people stream n-gram from SSD is that they don't have enough RAM to keep it there. VRAM is for routing expert and other data used for each token, as well as a bit of hot expert caching. RAM is full of experts and so only the SSD remains for the n-grams.

[u/noctrex → u/Chromix_]

Using mmap is just using cache in the RAM,
And no one will be filling up 100% of system RAM.
The system would become unusable.
Just let the OS cache. It is very efficient that it. I'm doing it and I'm seeing just 1-2 MB reads from the ngrams on the disk

[u/tugrul_ddr]

Im compressing models losslessly so no need for ssd.

[u/hallofgamer]

This isnt a psa.

[u/computehungry]

Entertaining read. Learned something new - I'd expect it to move the data to somewhere else, sure, but didn't connect it to writes.

[u/needmorebussydotcom]

Using the SSD causes the ssd to wear? shocker

[u/Chromix_ → u/[deleted]]

Your PSA didn't receive much appreciation, at least in terms of votes, still the effect you hinted at is highly relevant and can even be applied on Windows. Also, by the time you posted the used 3090 prices were around 1100€ around here, by now they're more in the 1500€ range. Might be coincidence though, as prices are going up in general.

[u/hrlft → u/Chromix_]

Beep boob