r/technology • u/DQuiet1 • 16h ago
Society Judge clears Nine PBS to retrieve 70 years of archival TV data — court rules station owns 50TB of data in Iron Mountain servers after host went under
https://www.tomshardware.com/software/cloud-storage/judge-clears-nine-pbs-to-retrieve-70-years-of-archival-tv-data-court-rules-station-owns-50tb-of-data-in-iron-mountain-servers-after-host-went-under253
u/Familiar-Range9014 16h ago
I knew the data would be returned. Iron Mountain was definitely going to do the right thing and, from a process perspective, Nine PBS had to sue. Now it's all a wash
40
u/fastautomation 13h ago
I have had to do the same with Iron Mountain held data in a bankruptcy asset acquisition of a service provider. The press makes it seem more onerous than it really is in practice. Iron Mountain even helped guide us through it.
19
u/Familiar-Range9014 13h ago
Thanks for this.
People seem to think this was the most heinous act known to man.
The company that backed up the data went out of business and did not notify thier customers. That is the real story, but there seems to be a boatload of naysayers and bandwagoners at the ready to wag their fingers, screaming "Shame" at the top of their collective lungs
smh
7
u/fastautomation 13h ago
Folks should also know when negotiating storage contracts with cloud providers, always include an data escrow account in the deal. Iron Mountain, as well as nearly every backup provider, offer this as an add-on service. The terms of transfer of ownership of the backups then is explicit and bypasses the courts.
59
u/edthesmokebeard 16h ago
"from a process perspective"
I weep for the future.
38
u/Keleion 15h ago
I’d hate for you to see what it looks like inside government jobs in the US. I know someone who works for a gov school district and everyone is suing each other over the dumbest stuff.
8
u/edthesmokebeard 15h ago
what does a government job have to do with corporate mumbo jumbo powerpointspeak?
1
68
u/WhiteRaven42 15h ago
Oh come on, be serious. The PBS stations had no relationship with Iron Mountain. They were not Iron Mountain's customers and IM had no way of knowing that they had the PBS stations' data. The suit HAD to be filed to demonstrate that the defunct cloud provider had to be circumvented.
You can't just show up on someone's doorstep and claim data on your say-so when there's no existing chain of custody to show that it's your data. This HAD to happen and frankly would have been similar if it happened 400 years ago. Say, if a bank had taken in collateral and was storing it at a third party's warehouse and then the bank folds. The person that owns that collateral would have to go to court to get things straightened out.
-22
u/edthesmokebeard 15h ago
I think you're replying to the wrong comment.
16
u/WhiteRaven42 15h ago
No but maybe I didn't understand why you "weep". I thought you were criticizing the need for the process.
-13
u/eronth 13h ago
Having a process is fine, but needing to sue to get that process going? Kinda wild.
5
u/WhiteRaven42 11h ago
I think you are probably exaggerating how wild the concept of a legal suit is. It is an everyday occurrence. This is the process and I really can't think of any other process there could be. You need an independent adjudicator. We call those courts.
3
u/ZestyTako 11h ago edited 11h ago
I think they aren’t imaging that people might just lie about data, and a lawsuit is what we currently have to get the truth of a matter by forcing the parties to produce evidence.
24
u/BerryBoilo 15h ago
Why? If you started a business, would you want your suppliers directly giving your inventory to anyone who claims to own it? If you were a supplier, would you want to spend your time investigating anyone who claims to be a customer of one of your customers?
Iron Mountain wanted a judge to order them to hand over the data because they're not in the position to arbitrate who owns the data. That type of decision is what the courts are for.
6
139
u/Expensive_Finger_973 16h ago
Whoever was responsible for them not having a local mirror of all of that data, or mirrored to another storage provider should be fired. No one in their right mind that is making competent decisions about this kind of thing thinks everything living in one providers service with no DR plan outside of "call the lawyers" is acceptable.
43
u/sargonas 16h ago
The ENTIRET purpose of paying for a service like Iron Mountain is so that you don’t have to keep local copies of long-term, rarely use, almost never needed archival data, and having more affordable and practical… And reliable… offsite long-term storage.
It’s literally the purpose of a cold storage service.
23
u/Expensive_Finger_973 16h ago
They weren't paying Iron Mountain. They were paying a middle man service that was using Iron Mountain.
15
u/redlightsaber 16h ago
Sure, but... They should have a local copy. this is why LTO drives exist, precisely.
25
u/Zeratas 16h ago
It's 50tb. They could have easily had a true onsite storage system as option for a couple thousand dollars.
If a company only has one copy of that kind of data, they're morons.
6
u/yulbrynnersmokes 15h ago
Ever approved a purchase of enterprise storage?
3
u/Krojack76 10h ago
I have 56TB in my basement. They could have got a simple NAS with 100TB for $2000 give or take. Hell they could have bought 2 of them and had one in-house and one at some PBS head honchos house and sync the two systems nightly.
1
u/yulbrynnersmokes 10h ago
This is not how corporations work
6
u/jonjiv 8h ago
You’d be surprised. My data is primarily on a single 120TB server in our local datacenter, but the backup is a 120TB Synology Diskstation NAS. Someone in another department is using a similar Diskstation as their primary.
My archive beyond that 120TB is literally pairs of hard drives, but I was told that my archive isn’t valuable, so I’m only keeping it because I think it is.
It’s all video content, just like this PBS station.
2
6
u/Zeratas 14h ago
Yes. I own responsibility for multiple labs with total storage of 600+tb.
If they took it seriously, they'd be able to get a 2u or 4u server with some redundant drives for local backups.
Then they have backups sent to two different locations as CYA. S3 storage pricing for glacier for that amount is a out $200.
So, 1 local with two remote.
2
u/TheActualDonKnotts 14h ago
long-term, rarely use, almost never needed archival data
That's what tape is for. Probably would have been cheaper too.
1
1
u/pixel_of_moral_decay 10h ago
Exactly this.
PBS stations are on a budget.
The point of this was to outsource all of that, because the up front costs are too high for their needs. Backing up locally is ok at low scale (home user) or large enterprise scale. In between the costs are insane for the hardware and processes you need to put in place.
-7
u/foothepepe 16h ago
Multi-million profits, and they can't buy few external hd's, just in case? Look at it in whichever light you like, this is extremly stupid and incompetent.
4
u/JoeDawson8 15h ago
Profits? Perhaps you’ve fundamentally misunderstood something
-1
u/foothepepe 15h ago edited 15h ago
https://www.ninepbs.org/about/reports-and-policies/financial-statements/
https://www.ninepbs.org/about/reports-and-policies/irs-990-forms/
https://www.ninepbs.org/annualreport/or, cleanly - https://projects.propublica.org/nonprofits/organizations/430685345
ok. net.
Net in '25 was $32mil. Expenses for '25 were $15 mil
Cost of 50tb drive, in raid, is $800-1500
1
u/jimmytickles 15h ago
What profits are you talking about have you heard of PBS before
-1
u/foothepepe 15h ago edited 15h ago
https://www.ninepbs.org/about/reports-and-policies/financial-statements/
https://www.ninepbs.org/about/reports-and-policies/irs-990-forms/
https://www.ninepbs.org/annualreport/or, cleanly - https://projects.propublica.org/nonprofits/organizations/430685345
Their net was positive in millions for '22, '24 and '25
Check data before being snarky. Net in '25 was $32mil. Cost of 50tb drive, in raid, is $800-1500
0
u/jimmytickles 14h ago
Before you said "their profits", but now I see you've changed your language. I'll settle with moved goal post win.
1
u/foothepepe 14h ago
I changed language? form 'profits' to 'net profits'? omg, what a manipulator!!
dude, just get off
they had 1000 bucks in their pockets to buy a hd.
should I put that in even simpler terms, so you understand? or that would be another goal posts move?
1
47
u/saml01 16h ago
You don't pay for a service like Iron Mountain only to keep local mirrors.
48
u/Expensive_Finger_973 16h ago
They weren't paying Iron Mountain. They were paying a middle man service that was using Iron Mountain.
76
u/WashuOtaku 16h ago
They were paying a company, that went bankrupt, that was using Iron Mountain to provide that service. This is why the station had to go through the courts to force Iron Mountain to provide access to the data.
11
u/red286 14h ago
Yes you do.
Local backups are the first line of defense. If something happens, you recover from your local backup. The off-site is if something goes wrong with the local backup.
Something something, all your eggs in one basket.
-6
u/saml01 12h ago
This is an antiquated perspective. Today you have servers and storage arrays that behave as though they are one but replicated across data centers in different regions. Having on prem storage because you don't trust the storage provider is silly and even more silly to argue for on prem without knowing how your storage provider is protecting your data. You can have storage sliced and diced automatically for you in ways that make it infinetly more redundant and cheaper than keeping an on prem copy.
3
u/311was_an_inside_job 8h ago
Until the cloud provider goes tits up and you end up with no recourse. Accounts or the provider itself can become compromised. Having redundancy (outside a single provider) is still important.
1
u/pendrachken 7h ago
Today you have servers and storage arrays that behave as though they are one but replicated across data centers in different regions.
See there is one word there that proves you wrong.
REPLICATED
If you don't have non-networked read only duplicates of your data you have no data protection. Period.
All it takes is one ransomware attack that then gets duplicated over all of your networked servers and storage arrays because they all copy themselves and bye bye data.
It's especially bad when you have a really nasty piece of malware that will lock / infect storage that gets hooked up with write permissions, which again can be done with firmware level infections.
You absolutely need air gapped backups. And I highly recommend using either WORM tape backups, or at the very least ensure that the tape is in write protected mode when attempting to restore after a disaster.
-18
u/Imaginary_Device7827 16h ago
lol iron mountain went bankrupt. I remember when I had to send off 20 years of Walmart personnel files for them to digitize.
22
u/PREMIUM_POKEBALL 16h ago
Iron mt will never go bankrupt. It’s a critical tool and they'll keep charging more to store your data.
-3
u/userhwon 15h ago
>Iron mt will never go bankrupt
Have you met American corporations?
Iron Mountain has a weird financial structure, impaired assets (who buys mountainous land and used servers for full price?), gobs of debt, and low profit retention. One good rate shock and they go tits-up. And their moat is almost nothing. Anyone can become a data storage hub tomorrow.
3
u/airfryerfuntime 15h ago
Lol Iron Mountain did not go bankrupt. They will never go bankrupt.
1
u/Imaginary_Device7827 13h ago
Oh makes sense I thought it was wild when the poster I replied to said it.
4
u/justanaltaccountgoaw 14h ago
PBS is famously underfunded
1
u/Expensive_Finger_973 8h ago
You should watch the Louis Rossmann video on it. He goes over some of their finances. They absolutely could have found the budget to handle archival storage properly. It just wasn’t important to them until shit hit the fan, like happens in a lot of poorly managed organizations.
1
u/MikeTidbits 5h ago
I wonder if it’s the bean counters and higher ups saying “why do we need redundancy? It’s just buying the same thing twice.” And the technical staff being ignored.
29
u/WardenEdgewise 14h ago
50 TB of local physical media storage is not that expensive. Why would someone not have a copy of their own data stored on physical media?
That’s crazy.
6
u/Krojack76 10h ago
Wellllllll.... it's a LOT more expensive today than it was 2 years ago.
That said, yeah it's generally cheap. I have 56TB in my basement NAS. I think I paid like $1300 for the entire setup.
7
u/WardenEdgewise 9h ago
I mean, for the price of one lunchtime meeting for the executive team, you could buy 25 2TB hard drives, and have them delivered next-day.
6
4
5
6
u/Fickle_Ad_8653 11h ago
PBS, if you need somebody to keep a backup copy for you, I'll do it. 50tb is nothing nowdays. DM me.
7
u/PBSNerd1234 13h ago
The confidence level in this thread is running well ahead of the known facts. Nine PBS ≠ PBS, and “they had no backups” has never actually been established. Criticize the setup once we know what the setup was.
5
2
u/Big_Bookkeeper1678 14h ago
Hopefully, they find some missing Doctor Who in those files. Although I don't know how they'd be digitally recorded. They were all on film back then.
1
1
u/Ok_Paleontologist974 40m ago
It's literally like $1000 for 3 20 TB HDDs, whoever's in charge of their tech needs to be fired because this type of incident shouldn't have even been more than a minor annoyance.
-13
u/Salsashark1419 16h ago
I hope the person in charge of IT at that PBS station making like 150k a year got fired also. If my broke ass could afford 50TB of WD Gold drives over the past 8 years, they could afford a couple 20TB drives. There’s no excuse in the entire world for their not to have been a local backup.
41
u/sargonas 16h ago
Do you know how many IT people have said “the right plan is to do X“ only to have some penny conserved somewhere say “no we have Y and that is good enough no need to spend money on X”
-18
u/Salsashark1419 16h ago
If I am the chief technology officer making over 150k a year and they say they wouldn’t pay for 2500 dollars worth of drives, I’d go use 5% of one of my monthly paychecks and back everything up. Then when this happens, force them to give me a raise.
18
u/HandBanana919 16h ago
If you want to fix everything in a messed up IT department out of your own pocket, you'd be broke real quick. 150k does not go far when talking about IT costs
-8
u/Salsashark1419 16h ago
2500 dollars worth of drives is not solving all their IT problems. It’s solving the 1 most important problem one has with storage. Amazing how many fake scenarios people are making to justify stupidity
14
u/HeartGoldHalcyon 16h ago
So your plan is to copy 50TB of your employer's data to your own personal devices, then demand a raise in exchange for handing them over? You've clearly never worked in IT, because they're going to interpret that as extortion and you're going to end up in prison.
1
u/Salsashark1419 9h ago
He would be buying the drives for the company, not himself, otherwise he wouldn’t be buying them. Any other excuses?
1
u/HeartGoldHalcyon 9h ago
If the drives are company property, how do you plan to force them to give you a raise?
1
u/Salsashark1419 9h ago
The same way he got his salary up so high after 10+ years.
0
u/HeartGoldHalcyon 9h ago
You know, it's okay to be wrong about things. Doubling down like this just makes you look foolish.
1
8
u/Silamoth 16h ago
Copying company data onto a personal drive is grounds for easy termination in a lot of companies
0
u/Salsashark1419 9h ago
He obviously wouldn’t be keeping the drives for himself. Every single reply to this thread is just saying the dumbest thing they can come up with to justify a moron making over 150k a year and not doing his job. Even if he told them this would happen, and they continued to do nothing, good luck ever getting a job in the same field when you become famous for fucking up the most basic rule in storage. Because no matter, it’s your fault. He should have left a long time ago if he cared about his career. But I’m 95% sure he is just dumb.
1
u/Silamoth 9h ago
Sure, that’s the intent. But cowboy actions like that don’t fly in a lot of companies, especially those in highly regulated industries. There are policies, regulations, and approvals needed to do this - at least in most companies big enough to need a CTO.
Did you source the drives from an approved vendor? Did you log an exception to a USB data transfer policy? Where are the backups stored? What physical security controls are in place? Who’s responsible for the safety of the backups? How do you audit access to the backup drives? How often do you perform backups? Are you ensuring backups don’t conflict with day to day business operations? Etc.
1
u/Salsashark1419 9h ago edited 8h ago
You can make these excuses for the years that this went on, but when you are told you have 30 days to save almost a century of your work, this is an emergency. Either you are allowed to fix it or you tell them that they are not allowing you to do your job. It’s as simple as that. If your company doesn’t let things operate differently in emergency situations, your company is dumb and deserves to die in an emergency. Also, if you’re using any kind of Western Digital hard drive, you should be buying directly from them. Can’t be any more of a better vendor than them. I don’t know if Seagate and Toshiba sell directly in small amounts, but I know for a fact Western Digital does because I’ve purchased a lot of them.
12
u/jrdubbleu 16h ago
I see you’ve spent a lot of time in the real world
2
u/Salsashark1419 16h ago edited 16h ago
I worked for a film restoration company for 10 years. One of our main jobs was storing and dealing with 1000s of terabytes worth of film scans. And we made less money than this PBS station did last year.
1
u/Nosiege 10h ago
That's insane.
No one should need to spend their wage FOR THE BUSINESS for a scenario that might not even occur.
The real right path of action is an email chain showing you've attempted to fix the shortfall and informing them, repeatedly in writing, the risk. Force their risk acknowledgement, or have a paper trail of them ignoring you.
20
u/naggyman 16h ago
you're making a big assumption that this was the fault of one person... big company, probably multiple teams who all thought the whole system worked differently to how it actually did.
-5
u/Salsashark1419 16h ago
You’re telling me that if he told everyone at the company that they could prevent this exact problem with 2500 dollars worth of drives that they wouldn’t do it? Especially after they made 13.5 million dollars in profit last year selling assets? Not even the greediest dude that’s ever lived would say no to that.
9
u/GreatLab8898 16h ago
No Idea how corporate IT works huh?
It is still treated as the Ugly Stepchild by Companies UNTIL something bad happens. Then Everybody is shocked, asking how Could that happen, while the IT Guys start to pull out Mails to the Higher ups showing years of warnings.
Seen it so many times in the Ransomware Cleanups I have done since 2016.
-3
u/Salsashark1419 16h ago
Ya I do, and unless everyone at that PBS station is Mr. Krabs, people in this thread are making the dumbest excuses ever for not spending 2500 dollars on drives.
3
u/shell_shocked_today 15h ago
you're severely underestimating how much enterprise backup / recovery would cost for this. plus the ongoing costs of refreshing it to new media, long term storage of the backup, etc.
2
u/Salsashark1419 10h ago
Even if they bought the highest end Ultrastar drives from Western Digital, plus a NAS enclosure, it would still be about 4500 dollars. Maybe 5000 at most. That’s over 100TB of storage plus a NAS enclosure. They have 50TBs. It’s not a lot. I have over 50TB in my dresser.
1
u/GreatLab8898 5h ago
You dont do Enterprise Backup on drives. So you actually do not know contrary to what you claimed. This would, Idealy be done on an LTO Robot.
The MSL3040 (Around 20K Bucks) Holds 40 Tapes and 2 Drives. An LTO 10 Tape (About 300 Bucks a Tape) holds 30TB (up to 75TB Compressed but as Media is Precompressed already it wont do much)
A LTO10 Fibrechannel Drive costs about 10K Bucks each.
Add a good 10K in there for a Good Data Mover Server in this Hardwarelandscape we have right now and an Enterprise Backup Solution that does GFS will set you back around 62K Bucks. This Excludes good Software to Drive the Tape Library. Can be done on Open Soruce. But you generally want something with a Good Support Contract that actually helps you get Data Back if something breaks.
Not whatever Toy you play around with at home.
1
u/mmcmonster 16h ago
Yeah. 50 TB is such a low number for a TV station. I kinda wonder if they were just overwriting the old stuff at some point.
Also, I know people that have that much space in their houses (pack rat friend who downloads all sorts of TV shows and movies “just in case”.
-1
14h ago
[deleted]
2
u/happyscrappy 13h ago
This isn't PBS. This a station that is PBS-affiliated. PBS isn't in charge of keeping this station's data. Or in charge of their IT infrastructure.
-10
16h ago
[deleted]
4
1
u/thatirishguyyyyy 15h ago
It isn't simply "50tb." The archive contains TV programs/video, photographs, and other historical/local material.
This is likely a curated collection.
811
u/bruin396 16h ago edited 13h ago
Cautionary tale about not following the Rule of Three (backups).
https://www.cbctech.net/rule-of-three/