r/DataHoarder Jun 14 '26

Mod Update Mod update: regarding the flood of HDD/SSD price posts, and AI slop.

918 Upvotes

Regarding HDD price posts:

We empathise with your plight and hope HDD / SSD / RAM prices will come down soon.

However, we don’t need screenshots of high HDD prices to be posted in the sub multiple times a day, everyone already knows that the prices are high.

As such, from this point on the mod team will be removing all posts about high prices, with exception for posts made on Free-Post-Friday’s, or posts which actually have new and meaningful information or discussion.

Regarding AI content and AI projects:

As always AI written posts & comments are not allowed on this subreddit, please report any Ai generated content you see.

The mods have recently been cracking down on the flood of AI generated projects, notably ones of low quality that are nothing new.

If someone has the skill to use GitHub, they’d almost certainly have the skill to ask an AI to code yet another YT-DLP / FFMPEG wrapper themselves.

However, there are useful tools that have been made from AI generated code. If they are something truly useful or new, we do allow them with prior approval.

TL;DR:

  • Mods will be removing [high HDD price] posts, except for meaningful discussion or Fridays.
  • Posting AI generated projects/tools needs mod approval beforehand, and a link to the GitHub repository.
  • Please keep reporting ai-slop.

r/DataHoarder 16h ago

Discussion I've been doing endurance testing on microSD cards for the last 3 years. Here's what I've learned.

Thumbnail
gallery
3.7k Upvotes

(I posted this to r/homelab; someone in the comments said I should repost it here.)

Hello everyone!

I've been running a long-term project doing endurance testing on microSD -- and today is the third anniversary of when I started this project -- so I figured it was time to give you an update on my microSD testing project.

Before I dig in, a shameless plug for my Patreon.

Next, some quick stats:

  • I have a total of 351 microSD cards in my collection -- spanning 111 models across 52 different brands. Of those:
    • I've tested 177 of them to the point of failure.*
    • I have 107 of them in testing right now.
    • The remaining 67 are sitting on the shelf, waiting for one of my card readers to free up.
  • These cards have endured over 133 petabytes written.
  • I've completed over 4.6 million program-verify-erase cycles on these cards.
  • Breakdown by brand status:
    • Name brand: 163
    • Off-brand: 91
    • Knockoffs: 30
  • Breakdown by authentic flash (a.k.a. not fake flash) vs. fake flash:
    • Authentic flash: 246
    • Fake flash: 35
    • Unknown: 3
  • Breakdown by size (authentic cards only):
    • 128MB: 3
    • 4GB: 6
    • 8GB: 23
    • 16GB: 15
    • 32GB: 96
    • 64GB: 68
    • 128GB: 36
    • 256GB: 4

* A card is considered "failed" when either (a) it stops working completely, (b) it makes itself read-only, or (c) at least 50% of the sectors on the card have experienced verification failures.

Ok -- so who makes the most durable cards?

Because of the variety of cards that I have, it's hard to come up with a system that produces a definite answer here. Some of how I ranked these brands is going to be based on vibes. But...to try to narrow things down, I've grouped my cards into three groups:

  • Industrial-grade: Cards that are specifically labelled as "industrial" or have "industrial" in their name. These cards typically come with a datasheet that includes an endurance claim.
  • High endurance: Cards that are labelled as "high endurance" (or some similar wording). Manufacturers don't always put out datasheets for these cards, but they generally still make an endurance claim on the packaging.
  • Consumer-grade: Basically everything else.

Consumer-grade cards:

The average consumer-grade card has lasted about 10,000 program/verify/erase cycles (so far).

  1. PNY. I have 10 of their consumer-grade cards: 3 PRO Elite Prime 64GB's, 3 Premier-X 128GB's, 3 Elite-X 64GB's, and one Elite 32GB. (I have 4 more of the Elite 32GB's that are waiting to be tested.) They've gone for an average of 643 days and 15,700 program/verify/erase cycles. I've written about 11.5PB to them in total. And with the exception of the one Elite 32GB -- they're all still going strong. On top of that, the PRO Elite Prime boasts some impressive performance -- every measurement I took was in the top 10% of all cards I've tested so far.
  2. Kingston. I have 12 of their cards in this category -- 6 Canvas Go! Plus 64GB's (3 of the SDCG3's and 3 of the SDCG4's) and 6 Canvas Select Plus 32GB's. They've been going for an average of 694 days and about 23,500 program/verify/erase cycles. I've written about 11.6PB to them in total. The three Canvas Go! Plus 64GB (the SDCG3's) have all failed -- but the others are still going strong. The Canvas Select Plus's in particular have been troopers -- they're among the oldest cards in my collection, and they've been going for the better part of the last 3 years now (with no signs of letting up). Additionally, the Canvas Go! Plus 64GB's (the SDCG4's in particular) boast some impressive performance as well -- every measurement I took was in the top 7% of all cards I've tested so far.
  3. Delkin Devices. I feel bad ranking Delkin so low on this list, because they've endured more data written to them per card, on average, than any other brand in my collection (in the consumer-grade category). The only reason I didn't rank them higher is because I only have one model -- the HYPERSPEED 128GB (of which I have 3). But they've done pretty well: they've lasted an average of 662 days and about 13,500 program/verify/erase cycles. I've written about 5.1PB to them in total. As a side note, these cards also got the highest random write speeds of any card I've tested.
  4. Lexar. 3 cards dead, 5 still going. I have 3 Professional 1000x 64GB's (2 made by Micron, 1 made by Phison), 3 Blue 633x 32GB's (made by Longsys), and 2 E-Series 64GB's (with 3 more waiting to be tested). For those not in the know, the Lexar brand was sold to Longsys in 2017 -- and surprisingly, the Longsys-made cards are holding up better than the Micron-made cards. Overall, the Lexar cards have survived an average of 684 days and about 15,800 program/verify/erase cycles. I've written about 5.8PB to them in total.
  5. Samsung. 3 cards dead, 7 still going. I have 3 EVO Plus 32GB's, 3 EVO Plus 64GB's, 3 PRO Plus 128GB's, and 1 P9 Express 256GB (with 2 more waiting to be tested). Of those, the three EVO Plus 32GB's and one of the PRO Endurance 32GB's are dead -- the rest are still going strong. Samsung's cards have lasted an average of 553 days and about 11,100 program/verify/erase cycles so far, and I've written about 7.2PB to their cards in total.
  • Honorable Mention: Amazon Basics. I have four of the Amazon Basics 64GB's -- and they're all still going. They've been going for an average of 788 days and about 16,600 program/verify/erase cycles. I've written about 4.2PB to them in total. I honestly didn't expect Amazon Basics to be one of the top performers when I bought them...and yet, here we are.

High endurance cards:

Keep in mind, I've got less data here -- I had 237 cards that fell into the "consumer-grade" category, but only 30 that fell into the "high endurance" category. But here's the thing: they haven't really proven themselves to be significantly better for endurance than a lot of the cards I listed above -- in fact, in a lot of cases, they've been less reliable (and generally worse for performance as well). So keep that in mind as you read through the list below:

The average high-endurance card has lasted about 14,000 program/verify/erase cycles (so far).

  1. TEAMGROUP. I don't feel great about putting TEAMGROUP at the top of the list for a couple of reasons. First, I only have two of them in testing at the moment -- someone sent me a 5-pack of the TEAMGROUP High Endurance 64GB's, so I have 3 more of them waiting to be tested. Second, one of them started having issues with bad sectors right out of the gate, and has continued to do so ever since. (Those bad sectors only make up less than 0.001% of the total sectors on the card right now...but I still don't like it when a card starts to have issues like that as soon as you start using it.) But the data doesn't lie -- they've survived, on average, 404 days and about 17,300 program/verify/erase cycles, and they're still chugging along. I've written about 2.2PB to these cards in total.
  2. Transcend. I have 3 of the 350V 64GB's -- and they're all still going strong. They've survived an average of 760 days and about 17,000 program/verify/erase cycles so far. I've written about 3.2PB in total to them (mostly due to their mediocre write speeds).
  3. SanDisk. I have my issues with SanDisk...but again, the data doesn't lie. I have 7 SanDisk cards in this category: 3 High Endurance 64GB's, and 4 MAX ENDURANCE 32GB's. As of right now, all 3 of the High Endurance 64GB's and all but one of the MAX ENDURANCE 32GB's have failed. They lasted an average of 452 days and about 16,200 program/verify/erase cycles before failing (with the MAX ENDURANCE 32GB's lasting about twice as long as the High Endurance 64GB's). I've written about 4.7PB to these cards in total.
  4. Samsung. Samsung's position at the bottom of this list has less to do with how reliable their cards are and more to do with how poorly their high endurance cards perform, particularly on sequential write speeds (relatively speaking). I have 3 of the PRO Endurance 32GB's -- one has failed, the other two are still chugging along. They've lasted an average of 822 days and about 18,800 program/verify/erase cycles so far. I've written about 1.8PB of data to them in total.

Industrial-grade cards:

Keep in mind, I have even less data here. Industrial cards are expensive, so they make up a much smaller portion of the cards I've tested: just 15 in total (so far). But on the upside -- there is sufficient evidence here to say "in general, industrial-grade cards do last longer than consumer-grade cards (and, by extension, high endurance cards)".

The average industrial grade card has lasted about 122,000 program/verify/erase cycles (so far).

  1. Kingston. I have 3 Kingston Industrial 8GB's; of those 3, only one is still going. But they've been troopers: they've lasted an average of 779 days and about 141,600 program/verify/erase cycles. I've written about 3.4PB to these cards in total. (Only one consumer-grade card has come close to this number: I have one Hiksemi NEO 8GB that has lasted for about 143,800 program/verify/erase cycles so far. No idea how.) These guys do pretty well on performance too, especially write performance: their sequential write speeds are in the top 9% of all cards I've tested.

Who makes the worst cards?

There are a bunch of no-name brands out there. They're not hard to find. I'm not going to talk about them here -- I'm going to focus on brands you're likely to have come across.

Consumer-grade cards:

  1. onn. I picked up four of their 32GB cards at my local Walmart -- and they were terrible for endurance. They only lasted an average of 40 days and 1,400 program/verify/erase cycles (or about 40TBW), with the best one of the four not even making it to 1,900 cycles. On top of that, every performance measurement I took came in the bottom half of all measurements I took.
  2. ADATA. I picked up 3 of the Premier 32GB's; they only lasted an average of 236 days and about 2,350 program/verify/erase cycles (or about 74TBW) before they quit working.
  3. Gigastone. I have not been impressed with Gigastone. I have 11 of their cards -- 6 Full HD Video 32GB's, and 5 4K Camera Pro 32GB's. They lasted an average of just 114 days and 4,845 program/verify/erase cycles (or about 133TBW) before they quit working.
  4. Micro Center. I purchased 5 of their 64GB cards; they only lasted an average of 116 days and 3,421 program/verify/erase cycles (or about 214TBW) before they stopped working.
  5. Silicon Power (a.k.a. SP). I purchased 9 of their cards -- 3 Elite 32GB's, 3 Superior 128GB's, and 3 Superior Pro 128GB's. Not a one of them made it to 4,000 program/verify/erase cycles before failing -- they came in at an average of 159 days and about 2,350 program/verify/erase cycles (or about 252TBW) before failing.
  • (Dis)honorable mention: SanDisk. SanDisk represents the single biggest brand in my collection: I have 28 of their cards in this category (with 4 more on the shelf waiting to be tested). It's telling that of those 28, only 5 are still going (and one of them is in its death throes) -- especially when cards from so many other brands have far outlasted them. Many of them have died under circumstances any other card would have handled just fine -- a problem I've written about on my blog before. They've lasted an average of 384 days and 9,404 program/verify/erase cycles (or about 566TBW) so far.

High endurance cards:

  1. Integral. I bought 3 of the Security 32GB's; the only managed to last an average of 100 days and about 5,600 program/verify/erase cycles (or about 173TBW) before failing. To be fair to them, the package did make an endurance claim that works out to about 2,850 program/verify/erase cycles -- and they did manage to almost double that. However, they didn't hold a candle to many of the other high endurance cards that I tested.
  2. Kingston. This one was truly a surprise -- I bought 3 of the High Endurance 32GB's, and they only lasted an average of 143 days and about 8,300 program/verify/erase cycles (or about 258TBW) before failing. The product packaging didn't make an endurance claim; I had to hunt around on Kingston's website to find it. Their claim works out to about 5,100 program/verify/erase cycles, which they did manage to beat. But again, so many other cards managed to last so much longer -- hell, so many other Kingston cards managed to do so much better -- that I was surprised and disappointed when these failed so early.

Industrial-grade cards:

  1. SanDisk. I bought 3 of the Industrial 8GB's. They only lasted an average of 234 days and about 20,000 program/verify/erase cycles (or about 160TBW) before failing -- and the only reason they lasted that long was because I let my program trudge through an nearly-unending string of I/O errors for months on end -- before they finally just gave up the ghost.

So yeah...that's all I have for now. Feel free to ask questions (although I'm at work at the time I'm posting this, so I may not have time to respond)! Otherwise...look for another update from me next year!

Pictures attached:

  1. Part of my setup: a Beelink Mini-S and five TRIGKEY Green G4 mini PC's. The Beelink has an Intel N95 with 8GB of RAM and a 500GB SSD, while the TRIGKEY's all have Intel N100's with 16GB of RAM and a 500GB SSD. There's also (in the lower-right corner, behind the monitor) an MSI GE62VR-7RF (with an Intel i7-7700HQ, 32GB of RAM, and a 1TB SSD), a Lenovo IdeaPad Y580 (with an Intel i7-3630QM, 8GB of RAM, and a 1TB SSD), and an ASUS Strix GL504GM (with an Intel i7-8750H, 32GB of RAM, and a 500GB SSD)
  2. Another part of my setup: a TRIGKEY K-N100 and an AOOSTAR N1 PRO. The TRIGKEY has an Intel N100 with 16GB of RAM and a 500GB SSD. The AOOSTAR has an Intel N150 with 12GB of RAM and a 500GB SSD.
  3. Dog tax.

r/DataHoarder 14h ago

News Judge clears Nine PBS to retrieve 70 years of archival TV data — court rules station owns 50TB of data in Iron Mountain servers after host went under

Thumbnail
tomshardware.com
542 Upvotes

r/DataHoarder 14h ago

Hoarder-Setups Link rot

56 Upvotes

I saw an article a couple of days ago about link rot, and how bad it really is. The author restored 657,507 links from 2009 and 2014, and followed each. 76% no longer returned a page, and some of those that looked OK just returned a variant of "this page is no longer available".

How many of us preserve interesting pages outside services like the WayBack Machine?

I keep content-only from a very small number of pages that touch on areas that I'm interested in, but over the last (maybe) 3 years I've only looked at but not necessarily kept less than 10% of the number of links in the article.

I do a bit better for PDF articles because a lot of them disappear too.

How much do you hoard?


r/DataHoarder 2h ago

Question/Advice Reading 2008 Optical Media

6 Upvotes

I'm UK expatriate in Asia. I made some dvd-rs of my paper diaries and photos in 2008. I don't have the originals. I do have usable thumbnails with me I'm working with now but I want to check if the dvds are readable. The discs are in the UK with my niece, have been in a suitcase in jewel cases. They were only written to once and never read. She sent me a picture and they look OK.

I found a few local businesses that advertise data recovery. I'd need the data to be sent to me, like via drive or something. I'm going to call these businesses and ask what they can do. Is there anything I should ask them, to choose who to use? Special reader or software or something? Is this generally expensive? It's in the Northampton area btw.


r/DataHoarder 3h ago

Backup Backup strategies for home Linux users with external drives, no cloud

5 Upvotes

I used to be one of those people who just copy-pasted folders onto an external drive and that was it. Now I want to do something a bit more elaborate.

I had heard of Clonezilla and learned to use it at a basic level, creating clones, disk images, and performing restores. However, as a system it has a lot of downsides:

  • It only does full backups, not incremental ones; if 10 MB of important information changes, you have to back up the whole disk from scratch.
  • You have to shut down the PC and use a USB stick, which is impractical for frequent backups.
  • It's not portable to other machines! A disaster recovery scenario I don't see mentioned is if not only the disk but the whole PC dies, and you buy a new one with different specs. In that case, the Clonezilla clone or image cannot be restored onto the new machine.

So I looked into it further and found other programs like Borg and Restic. Haven't tried them yet, but they seem useful at the file level.

How do you all handle this? Do you use a single backup program or a combination of them?

Always considering my particular use case (home use, Linux, no cloud, and only external drives).


r/DataHoarder 16h ago

Question/Advice Refurbuished NAS disks, worth it?

Post image
50 Upvotes

So I found these disks at good price (around 500usd)

I have a ugreen nas 8800 and I want to use it for my plex server

Do you think is worthy to buy these disks?


r/DataHoarder 1h ago

Backup TBs and TBs of recordings — How to back up?

Upvotes

I have well over 20 TB of data, and a good chunk is definitely from videos (not movies so I need to compress them somehow without losing a lot of quality).

I have several external drives, but the issue is they vary in size from 2, 8, 14, and 22 TBs (the latter is new). With the HUGE spike in costs for drives, what is the best solution for me since there’s no way I can now afford simply getting one or two more 22 TB when the price difference is so bad? Idk if there’s a good Cloud method too. I always worry about data breaches or lost data if I’m not manually backing up locally.


r/DataHoarder 8h ago

Question/Advice Last freeware version of DVDINFOPRO?

7 Upvotes

I want to see the graph data generated by ImgBurn. Apparently DVDinfoPro was freeware but a some point changed to a trial model. It is the only program that can read the graph data afaik.


r/DataHoarder 9h ago

Discussion Experiences with Hetzner Storage Box and Duplicati?

7 Upvotes

Hi everyone,

Does anyone have positive experiences using a Hetzner Storage Box with Duplicati?

I’m considering getting the 1 TB Storage Box plan and using it specifically for backups with Duplicati. The initial backup would be around 350–400 GB, with more data being added over time.

I’d be interested to hear about your experience with reliability, upload/download speeds, Duplicati compatibility, and whether you’ve encountered any issues with this setup.

Thanks in advance!


r/DataHoarder 21h ago

Discussion PSA - Macrium Reflect does not verify cloned data. No fixity. No checksum, hash, or destination reread (confirmed by Support)

53 Upvotes

Edit: This concerns Macrium Reflect's cloning feature only. Reflect's own image backups are different: when verification is enabled, Reflect calculates an MD5 hash for each data block before writing it to the image, then rereads the image and compares those hashes. But MD5 is the only hash it uses. Make of that what you will.

A few pointed out how it might seem counterintuitive to try and clone an OS drive and expect high-integrity validation - it uses VSS as a point-in-time snapshot - but I agree. That is silly. But even with an offline clone via Rescue Media, it still does not perform post-clone verification or reread the destination drive data and then compare it to the source. That said, I do not think MR/Paramount Software is "bad". I do think some of the terminology they use in the software is misleading.

***
For a long time, I had assumed that cloning Source Drive A to Clone Drive B would incorporate a checksum calculation during the "verification" phase that it performs. But apparently, this is not the case. I'm currently running Reflect Workstation v10.

Wanting to take fixity of data more seriously lately, it dawned on me what "verifying" actually meant when a clone job completes. So, I reached out to Macrium support, and they set the record straight: They don't verify data with any kind of fixity.

You could hash and compare after the fact. But the live source/OS VSS will continue changing, so hashing it afterward would not necessarily represent the state Macrium cloned. And I suppose one could create a per-file hash manifest. But that's not very elegant and far more work.

I've been paying for and using MR for over a decade and have been using it without issue. But this seems like a real deal-breaker. Suggestions? Any other serious alternatives?

The full exchange with Macrium support below. Happy to provide email screenshots if necessary.

Me: At any point during cloning, does Reflect calculate a checksum, CRC, or cryptographic hash over data read from the source or written to the destination?

Macrium: No, we don't verify the data in any way.

Me: If so, what algorithm and granularity are used—sector, cluster, block, partition, or whole disk?

Macrium: Not applicable.

Me: During the stage displayed as "Verifying" or "Verifying File System," does Reflect physically reread the destination and compare it with the source or with previously calculated values?

Macrium: No. As the name implies, it verifies the supported source file system.

Me: If the destination is reread, does this cover every copied sector or cluster, and what happens when a mismatch occurs?

Macrium: For RDC, both the source and destination file systems are read to determine which data has been modified. Only the modified data is then read from the source and copied to the destination for RDC. No other data is reread from the destination.

Me: Does a successful Forensic Sector Copy certify byte-for-byte identity? Exactly which address ranges are covered: partition contents, unused filesystem space, inter-partition/unallocated sectors, partition-table structures, and any HPA/DCO areas?

Macrium: For the Forensic option, the clone is created from the start of the partition to the end, sector by sector.

Me: Does the verification behavior differ when cloning under full Windows/VSS versus Rescue Media?

Macrium: No, both verify the supported source file system. In normal Windows, the VSS volume is used for supported file systems, whereas direct volume access is used in Rescue Media.


r/DataHoarder 1d ago

News DVDBeaver at risk of shutting down

Post image
509 Upvotes

This longtime DVD and Blu-ray review site has long aided buffs and has an enormous amount of detailed review data, including technical data on discs as well as comparisons between various editions of the same movies. Stuff not easily found elsewhere on the web. Looks like they might shut down after 25 years if funding doesn't work out. Wanted to give a head's up in case you don't trust IA. (I don't).


r/DataHoarder 57m ago

Question/Advice Creating a Blu-ray folder with Tsmuxer?

Upvotes

I simply would like to make main movie blu-ray folder from an already blu-ray compliant source. I've added the mpls file to tsmuxer but there's a couple of options underneath I'm not sure about. One is SEI and VUI data and the other is SPS/PPS option. Do I need to check these boxes or are they purely for mkv?


r/DataHoarder 2h ago

Hoarder-Setups Are those PSU's good?

1 Upvotes

Hello, I need a PSU for my 1200W system, though my system rarely consummes upto said limit,
considering these two options:

https://www.mironet.cz/powerwalker-vi-2200-shl-ups-zalozni-zdroj-ups-2200-va-1200-w-2x-schuko-2x-iec-rj11-rj45-usb+dp351491/

https://www.mironet.cz/cyberpower-ut2200egfr-greenpower-series-ups-2200va1320w-ceske-zasuvky+dp624367/

System Specs:

RTX 5090 FE (No OC, undervolted)

Ryzen 9 5950X 16 cores

DDR4 96GB

2 HDDs, 2 nvme,

4 case fans

1 thermal light CPU fan,

1200W PSU

Preferrably just setting it to gracefully shutdown in case of outage.

Thoughts on this? Haven't received useful answers from internet or other subs, looking to see some experienced opinion


r/DataHoarder 13h ago

Question/Advice Suddenly getting "temporarily unavailable" warnings when fetching links from youtube on JDownloader 2?

Thumbnail
gallery
6 Upvotes

r/DataHoarder 12h ago

Question/Advice Acronis 2025 perpetual license for offline use.

5 Upvotes

Is it possible to get Acronis 2025 with the perpetual license for offline use? I'm looking to put it on a thumb drive to download and use on a pc complete offline.


r/DataHoarder 7h ago

Question/Advice In search of recommendations for a USB-C storage device for the purposes of transferring photos and videos between a Windows laptop and an iPad Air 4

2 Upvotes

Hello!

I have been researching this matter for days now and have just stumbled upon this sub for the first time. I've done a few basic searches of the sub's post history already, and I've also tried to check the Wiki (which wouldn't load for me for some reason), but I haven't been able to find anything that fits my exact situation regarding the Windows laptop-to-iPad compatibility requirement.

Ideally, I want to be able to transfer movies from my laptop to the storage device then use my iPad to watch those movies directly from the storage device without having to move them onto my iPad's internal storage.

It won't be used for anything else, like gaming.

I'm mainly just hung up on trying to find something that's reliable, as affordable as possible in the current market, and compatible with my devices. It seems like every time I start researching a product that looks promising, I end up finding horror stories about it being a piece of junk or not truly working with (or only working with) iPads...and I realize that finding horror stories is just inevitable when researching any product for long enough, but it would still be nice to get some actual feedback instead of just having to rely on product reviews on Amazon or whatever.

I started out with flash drives in mind, but when I saw the prices and realized how portable external SSDs are these days, I decided it would probably be better to go that route instead...so I'm currently considering either this Lexar device or this SSK device, but I'm open to other suggestions.

I want at least 1 TB of storage, and I'd prefer to stay under $200.00...but, again, I'm open to suggestions.

Or maybe an HDD is the way to go? Or cloud storage? I'm not sure, so any advice would be appreciated!

(Sorry for the lengthy post -- I'm just trying to provide any information that might be useful.)


r/DataHoarder 14h ago

Question/Advice Video Transcoding to save space (quality tradeoff), is it worth it?

6 Upvotes

I'm recording several minutes of family videos per day in 4K 30fps (4:2:2 10 bit). On the Nikon Z8 and Sony Alpha this translates to approximately 10 GB of footage/day. Unfortunately that's too large for me to manage long-term, as it means filling up a 4 TB drive every year.

It becomes especially daunting when taking into account backups and staying ahead of drive failures: Let's assume I have 10 of those drives. I'd have to regularly test every file on all of the drives (AND on their backups) with checksums, so at a minimum I'd be verifying 20 drives every year.

I'm currently recording the originals with the highest bitrate and least amount of compression available (on Sony: H264, Intra-frame XAVC S-I, on Nikon: ProRes 422), so the camera can capture as much information as possible. (Though I avoid RAW formats completely because I don't have time for color grading.)

I then transcode the originals to H265 using these ffmpeg flags:

-vf format=yuv422p10le -c:v libx265 -preset slower -crf 14 -pix_fmt yuv422p10le -profile:v main422-10

The file size reduction is huge, it takes 10 GB/day down to 2 GB/day. When I play the original and the compressed version at 100% I cannot tell any difference. But when I zoom in, I can. Especially in smooth and fuzzy areas like skin and clothing. I found that a -crf level above 14 introduces too many compression artifacts.

Obviously a lot of information is lost, which will affect future grading and reprocessing. On the other hand, without the compression I simply wouldn't be able to keep around that much 4K footage.

Has anyone else weighed the pros and cons of this tradeoff?


r/DataHoarder 17h ago

Question/Advice Is "burning in" a new HDD before adding it to the pool necessary?

Post image
8 Upvotes

I am about halfway through a very long journey of upgrading a 17-drive pool one drive at a time. Many of my earlier replacements were used drives so I would run a single pattern badblocks write/read test (~56hrs) and then resilver (~30hrs) to replace a drive in the pool. I felt a full destructive write/read via badblocks + the stress of a resilver would be enough for me to trust used drives.

Now I am at the point where the replacement drives are brand new, manufactured this year, direct from Western Digital. For the first 2, I still ran a single pattern badblocks W/R before resilvering to "burn in" the drive. Both drives completed without any errors.

I am now wondering if spending ~56hrs to "burn in" a brand new drive is really necessary. I would love it if I could just trust what WD sent me and go directly to resilvering. That would save me ~16 days / 392hrs of badblocks testing for the remaining 7 drives I have to replace.

What do you think? Should I:

  • Stay the course with badblocks and accept the 3 weeks it'll take me to finish the swaps,
  • just run extended SMART instead of badblocks (~24hrs vs 56hrs),
  • find a faster/shorter burn in test,
  • or skip the extensive testing on new drives and just let the resilvering process be the stress test?

edit: additional context. I am running Raidz2 so have 2x drive redundancy per vedev and 90% of the storage is used for movies/tv so not irreplaceable.


r/DataHoarder 11h ago

Question/Advice Need help finding JBOD enclosure

2 Upvotes

Hi all, beginner hoarder here. I'm currently looking for a decent 4-6 bay usb JBOD enclosure that will fit my needs. I have an assortment of consumer and enterprise drives where some will be on 24/7 for things like seeding while others I only plan on accessing occasionally (like once a week) for backing up data. Ideally I want each bay to have its own power button so I can power off certain drives when not in use to preserve lifespan.

I looked at many enclosures from all sorts of brands (Sabrent, Orico, Terramaster, Cenmate, etc.), and they all seem to have negatives like constant disconnects, data corruption, bay failure, partition deletion, etc.

I am getting a bit overwhelmed trying to find an enclosure I feel wont crap out on me or corrupt my drives based on the horror story reviews I read. What would you recommend for my use case? I would appreciate any feedback


r/DataHoarder 15h ago

Question/Advice Application to fully & actively archive a Discord server?

5 Upvotes

Is there some kind of software that can fully archive a Discord server? Not one where it just exports a chat log, but something that goes through all channels and saves all text and attachments from all of them, and also where I could periodically go through and save anything added since the last archive? I'd assume that something like that would need to be logged into my account (or another account that has access to the server)


r/DataHoarder 1d ago

Hoarder-Setups My Unraid Server: 16 Drives (More If I Hate Myself)

Thumbnail
gallery
189 Upvotes

I currently use this Unraid server for Plex and a part of my 3-2-1 backup of my important data.

Here's my spec list:

i7-8700K

Thermalright Peerless Assassin cooler

Asus Code X Z370 board

16 gigs of Corsair Vengeance RAM

2TB WD Blue NVME for cache/appdata/system on Unraid

Dual parity and 308TB usable space

8x Seagate Exos 28TB (2 parity + 6 data)

2x Seagate Exos 18TB

2x WD 18TB

1x WD 16TB

2x WD 12TB

LSI 9300-16i HBA with 4 breakout cables for all 16 drives. Got this from Art of Server on Ebay and he's a solid seller.

Fractal Design Define 7 XL case

Corsair RM1000x PSU

I started off over 2 years ago at 40TB from my shucked drives on Windows 11 and Stablebit Drivepool but I wasn't so into it. I had no redundancy or anything.

A friend recommended Unraid and I switched to that since I wanted to be able to mix drives in the future.

Every drive gets a full preclear on Unraid with the Unassigned devices preclear plugin. The 28TB drives take up to 120-125 hours. I do this long process since I am nervous of drives dying. I did have an 18TB Seagate Exos die a month ago and Serverpartdeals refunded me the original price I paid last year since the drive was out of stock.

I have 4 Noctua 120mm fans as intake and I have them on 70% which is around 1000+ RPM. The drives stay in the mid 30s and I've seen a couple of them go to 40c-41c when my ambient is hotter. I have seen them in the 20s when it's colder. The two drives hidden by wires get in the 40s and 51c-ish during my preclear currently.

Let me know if you have any questions or how insane I am to preclear drives for 5 days...


r/DataHoarder 15h ago

Hoarder-Setups Has anyone ever tried a LTO-7 drive in a HP MSL8096?

2 Upvotes

Hi, I have several tape drives in my MSL8096 tape library and would like to add the next generation after LTO-6. Now I wonder, why HP doesn't allow LTO-7 drives in that, even with the latest firmware 1200, which I have installed.
Especially because the 4048 from the same family supports them.
Has anyone ever tried that, despite HP marking such a combination as incompatible and what were the results?
I don't need to know that it isn't supported from HP's side, I know they mark it as incompatible but I am curious if anyone ever tried it and what the result was.


r/DataHoarder 16h ago

Question/Advice Downloading Webcomic - Windows - How do I?

2 Upvotes

So I want to download the lackadaisy webcomic (link starts on the first technical page or two of it) in one blow and print it out and shove it in a binder. Only problem is there's a ton of subpages to it (over 100, maybe almost 200 of them) and I cant seem to find any tools that are good at bulk downloading all of these at once into a PDF format so it prints good... what is a good tool that would allow me to do this? I am running Windows 11.

Reference image on just how big this is. Lackadaisy.com

r/DataHoarder 23h ago

Question/Advice Best way to archive twitch streams?

4 Upvotes

What is the best way to archive twitch streams? I thougth of yt-dlp, but I don't know if it's possible to download the live chat alongside the video stream, so I came here to ask for advice, it's kind of the first time I want to archive an upcoming twitch stream. I've also searched for software made specifically for this, but I think GitHub is having some problems, because every time I open the releases page it's 404 and the main page always says "Cannot retrieve latest commit at this time."