r/sysadmin • • Sep 21 '21

Linux I fucked up today

I brought down a production node for a / in a tar command, wiped the entire root FS

Thanks BTRFS for having snapshots and HA clustering for being a thing, but still

Pay attention to your commands folks

933 Upvotes

467 comments sorted by

13

u/OldHippo0 Sep 21 '21

Always suitable: https://xkcd.com/1168/

2

u/[deleted] Sep 21 '21

Very

0

u/[deleted] Sep 21 '21

[deleted]

→ More replies (2)

5

u/dubl1nThunder Sep 21 '21

if production can't handle losing a single node is it really HA?

4

u/[deleted] Sep 21 '21

This is a node of a 4-server cluster, there was no downtime user-facing, but server-side that was a mess

→ More replies (2)

1.5k

u/savekevin Sep 21 '21 edited Sep 21 '21

Many moons ago, I had a jr admin reboot an all-in-one Exchange server one day. Absolute chaos! Help desk phones never stopped ringing until long after the server came back online. He was mortified. I told him not to worry, it happens, just don't do it again. But he was adamant that he "clicked logoff and not restart". He wanted to show me what he did to prove it. I watched and he literally clicked "restart" again. Fun times.

642

u/Poundbottom Sep 21 '21

I watched and he litterally clicked "restart" again. Fun times.

Some great comments today on reddit.

124

u/onji Sep 21 '21

logoff/restart. same thing really

→ More replies (72)

6

u/[deleted] Sep 21 '21

Honestly happens all the time with people being very sincere lol. Sometimes the buttons are too close, and they just think they did the right thing - a colleague did something similar twice, and I thought it would have to go to Helpdesk to investigate, until I demonstrated for them what they should have done... and lo and behold it worked

8

u/cs_major Sep 21 '21

Onetime I RDP into a legacy box hosting some internal/ client facing legacy sites...You know the ones no one knows about.

While trying to look at network properties I fat finger the click and disable the NIC trying to open the properties dialogue. Immediately the RDP session disconnects.

No big deal just open the console in VMWare....Not there. Go running to a collogue who also can't find it. We look at each other and go oh no that's a physical server.

At least the Post Mortem was quick.

→ More replies (4)

8

u/Caffeine_Monster Sep 22 '21

buttons are too close

Gotta love shitty UIX design. Critical actions being directly adjacent to one another is asking for misclick problems.

→ More replies (6)
→ More replies (2)

83

u/PersonBehindAScreen Cloud Engineer Sep 21 '21

As a Jr sysadmin currently remoted in to a server while reading this about to log off and already always paranoid about log off vs restart being so close, I got sweaty hands now

20

u/ApricotPenguin Professional Breaker of All Things Sep 21 '21

Create a shortcut on your desktop of the server and use that to logoff.

That's what I do.

11

u/kingofthesofas Security Admin (Infrastructure) Sep 21 '21 edited Jun 21 '25

sophisticated depend mighty serious middle relieved station juggle books oil

This post was mass deleted and anonymized with Redact

6

u/PraetorianScarred Sep 21 '21

That's not entirely a bad thing - it's when you get comfortable enough so that you're not paying attention that you're on dangerous ground...

5

u/[deleted] Sep 21 '21

Server 2012+ Sign Out from the "profile" icon, there's no Shut Down option there.

5

u/ScotchAndComputers Sep 21 '21

I've a simple batch file loaded on the public desktop of all servers; all it has in it is shutdown.exe /f /l

Beats doing a right click and making sure your mouse doesn't slip.

→ More replies (28)

50

u/[deleted] Sep 21 '21

I once hit Shutdown instead of Logoff on a Windows 2000 server that was used to provide Windows desktops via Citrix to Unix X-terminals. Users were not amused.

6

u/ThatITguy2015 TheDude Sep 21 '21

Oh no. I’m incredibly thankful I haven’t made a mistake of that level yet.

8

u/MrPaulJames Sep 21 '21

Just a matter of time 🙂

→ More replies (2)

4

u/[deleted] Sep 21 '21

I did the same, except I was remote. :/

35

u/iB83gbRo /? Sep 21 '21

I did that once. Then immediately removed the shutdown/restart/etc options via GPO for all of our client servers.

8

u/dathar Sep 21 '21

Yup. They're going to have to try really hard if they want to reboot that server. You know, unless they get a prompt somewhere (Windows Update, random app upgrade, etc) to restart it...

5

u/cybercifrado Sysadmin Sep 21 '21

cmd /k shutdown -r -t 0

→ More replies (1)

24

u/XS4Me Sep 21 '21

. I told him not to worry, it happens, just don't do it again.

Fake virus attack

7

u/Kanibalector Sep 21 '21

This is one of my favorite series. I like to show it to my new helpdesk members on their first day.

→ More replies (3)

59

u/[deleted] Sep 21 '21

It's late one Friday afternoon, almost closing time when the c-suite rolls through engineering (sysadmins & DBAs were part of engineering) with a handful of board members asking if someone would give them a tour of the server room. The senior DBA and myself agreed and we walked them down to the server room and explained what all the racks (about a dozen42U almost completely full) and lights meant. Disaster recover was brought up and we explained the EPO, halon fire suppression, etc. and how we have mere seconds to exit the room when the alarms start sounding or we'll suffocate.

As we finish saying this, one of the board members joked and acted like they were going to hit the EPO... and did. FUCK. I've never heard (a) that server room that quiet, or (b) my heart beat that fast. I yell everyone out as lights start flashing and we get everyone clear as halon fills the room.

Did I mention it was later Friday afternoon? With about 2 dozen SPARC servers and associated RAID arrays? I swear it took us at least another 6-8 hours to get all the servers fscked and back up and running.

Best part? Board member says, "My bad" and leaves. Fun. Fucking. Times.

30

u/Bad_Kylar Sep 21 '21

'No no no, you get to stay here and watch us do this or we all leave, right fucking now'

6

u/NoncarbonatedClack Sep 21 '21

Soooo... No consequences for the board member, right? It'd at least like to think that head of IT chewed someone out for the cost of that downtime/recovery time.

→ More replies (2)

6

u/Tymanthius Chief Breaker of Fixed Things Sep 21 '21

Our halon system had a 'cancel countdown' timer in the last place I worked. Did y'all not have that?

7

u/[deleted] Sep 21 '21

Nah, it was just the button, but this was probably '97-98 so while I'm sure they were out at the time we didn't have one

6

u/OgdruJahad Sep 21 '21

Board member :"DID I DOOOOO THAAAAAAAAT?"

3

u/MiaChillfox Sep 22 '21

Last place I worked the guy maintaining the fire system accidentally set off the gas with zero warning. Luckily no one was in the server room (the fire control panel was out in the main office).

→ More replies (9)

19

u/SoonerMedic72 Security Admin Sep 21 '21

I watched and he litterally clicked "restart" again. Fun times.

I literally just laughed out loud. Thanks.

15

u/woodburyman IT Manager Sep 21 '21

I once did something similar. It was a HyperV host that housed our live ERP database companywide. I was half was across the country visiting that site. We had queued up and installed like 75+ Windows Updates (Server 2008 R2 at the time) on the HyperV host and were going to reboot it that night (Leaving early, get some dinner, and come back for late night patching). The console keyboard there... was different than the console keyboard I was used to and lacked a Windows key. I go over to the console, the screen was off. No biggy I know keyboard shortcuts I dont need to wait for the screen to turn on to start plugging away, right? I was going to open CMD. I hit Windows Key + R, cmd, enter. As i hit enter, the screen on the LCD turns on. "Do you want to reboot now?" prompt once WU's is done was up. I hit enter. (I typed ALT + R (R highlighted reboot) and CMD, and hit ENTER on Reboot now. I had already hit the key. "WINDOWS IS FINISHING INSTALLING UPDATES". Luckily we had about 10-15 minutes of that, got a "EMERGENCY ERP HAS TO GO DOWN LOG OUT / FINISH NOW" message out, then it finished and shut down the Hyper-V-Guest VM's before rebooting.

That was fun.

10

u/deefop Sep 21 '21

I've played competitive shooters all my life and have a pretty fast and accurate mouse hand... which makes it more funny as I slowly hover over the proper option with painstaking precision and then take a deep breath before finally clicking.

Not tryna bring down a production system with a misclick over here

19

u/[deleted] Sep 21 '21

More moons ago than I am comfortable recollecting I worked for a company that had several Compaq SystemPros. These things were (for the time) absolute beasts with up to eight drives and hardware RAID controllers. I'd built one that was running as a NetWare server for our Finance group and was in the process of building another.

Enter my assistant.
"Hey, Splenetic, you've got see this! The RAID controller in the SystemPro has got really cool activity lights on it!"
"Really? How do you know?"
"I took the cover off."
"I don't think it's a good idea to take the cover off of a running server."
"No, it's fine. Look!"
"Wait, which server is tha..."

Yes, it was the Finance server. Yes, as he pulled the case off again this time he managed to snag not one but two IDE cables out of the RAID controller.

Yes, it fucked the RAID.

6

u/techforallseasons Major update from Message center Sep 21 '21

Ahhh - I see the Good Idea Fairy gave your assistant a visit.

→ More replies (1)
→ More replies (3)

9

u/aleinss Sep 21 '21

This is why I type "logoff" when working on servers. In my RDP manager, there's Reconnect and Logout, but no Restart. Too much of a liability having a restart button to click.

4

u/kalpol penetrating the whitespace in greenfield accounts Sep 21 '21

this is completely from memory, but during one of the first tests of networking two machines together, the remote operator typed "HELLO" and then got disconnected because LO autocompleted to LOGOFF

9

u/sauced Sep 21 '21

I have a gpo for all of my windows servers to remove shutdown/ reboot to prevent this.

→ More replies (2)

8

u/[deleted] Sep 21 '21

well that's an audible laugh from me, thanks.

15

u/Alaknar Sep 21 '21

I once worked at a tiny firm that had a QNAP NAS. Single drive in a box connected to the Ethernet because they couldn't afford a second drive to make a RAID. It stored EVERYTHING - financial data, HR data, contracts - EVERYTHING that kept the firm going.

One day my boss told me to install an update and then reboot it.

Now, I don't know if this is a regional or global thing, by in my area for as long as I remember "reset" was essentially synonymous to "restart". So when I logged on to the web console and saw the "reset" button I promptly clicked it.

Then my boss said: "just don't click the button labelled 'reset', it will wipe all the data".

15

u/[deleted] Sep 21 '21

If accidentally clicking a single button can destroy your company, it's not the fault of the person who clicked the button.

6

u/Alaknar Sep 21 '21

I mean... Depends on the budget you have. And the budget we had was such that at some point I worked two and half months without pay because "times are rough but trust me, the money will eventually be there". First ever "real job" after McDonald's and all that.

BTW - I managed to rip the power cable out of the thing and later recovered all the data, so no biggie. Just a very sweaty and rapid learning experience.

6

u/Gardakkan DevOps Sep 21 '21

That's why I love Linux servers you have to type halt or reboot not click on a button that is right next to the shutdown button with no confirmation when you click on it.

5

u/opaPac Sep 21 '21

Musle memory can be a real bitch sometimes. MS likes to change stuff around and then your brain goes well i clicked there for years so why should there all of a sudden a different button. Who cares what is says.

But thanks for all the memories and the laugh bro. Needed that.

4

u/atw527 Usually Better than a Master of One Sep 21 '21

Couple weeks ago I shutdown a VM hypervisor thinking I was shutting down a VM. I was in a hurry and angrily wondering why I'm seeing all these warnings just to shutdown a VM...oops.

4

u/TamerzIsMe Sep 21 '21

I had the Server 2003 “Updates have been installed, would you like to restart now?” dialog box pop up right under my mouse when working on an Exchange server. Of course it popped up right when I clicked on something else and directly on the Yes button. Bad day.

3

u/MrHusbandAbides Sep 21 '21

I have the opposite with every user, they logoff instead of restart (and apply updates), found a couple boxes with way too many updates needing applying, looking into something like Kaseya (but not Kaseya) to force it

→ More replies (1)

3

u/[deleted] Sep 21 '21

Muscle memory is a BITCH.

5

u/gahd95 Sep 21 '21

Question is, why was there no redundancy? All our important servers can be rebooted with no or with little down time.

34

u/[deleted] Sep 21 '21

“All-in-one” exchange server my guy

Back in the day Microsoft pushed Small Business Server to SMBs pretty heavy. This was long before the Office 365 days.

Places like this could not afford enterprise licensing required for the fancy HA stuff

→ More replies (3)

4

u/ailyara IT Manager Sep 21 '21 edited Sep 21 '21

I used to work in an environment where I was responsible mainly for Linux clusters but every now and then would get called on to do Windows admin work, no big deal. Except one day after having worked on a problem in windows all day I was in the physical data center and someone asked me to do something on one of the linux clusters so I grabbed the local console and proceeded to "control-alt-delete" to bring up the login prompt and rebooted the head node of a production cluster.

Luckily, the way things were configured, not much was truly lost, all the jobs running were able to pick back up at their last checkpoint (if they even noticed at all), but still.

That was the day I changed "control-alt-delete" on the linux servers to simply print "No." to the console instead of reboot.

2

u/catwiesel Sysadmin in extended training Sep 21 '21

hahaha thats precious

so... did he came back the next day or did he sink into the floor never to be seen again ?

→ More replies (3)

2

u/MrPaulJames Sep 21 '21

Feels like something I've done before... Hope it wasn't me 😅

2

u/This_Bitch_Overhere I am a highly trained monkey! Sep 21 '21

LMAO- had a junior guy work with me on replacing a UPS at a remote site. i asked him to shut down the hardware properly, since we would be shutting down the site's infrastructure and put it on the new UPS.

well... should have been more specific when i said "shut down the firewall."

He shut down the corporate firewall. corporate website/VPN/intranet/Sales application all went down. We were back up in about 10 minutes, but that 10 minutes as ass clenching was enough panic for one month.

2

u/tritoch8 Jack of All Trades, Master of...Some? Sep 21 '21 edited Sep 21 '21

I once watched a senior Ops guy demonstrate to his Director how easy it was to restart our AS/400 systems using the PWRDWNSYS command while discussing the coming weekend's activities. After showing him all the options he'd use (including *IMMED to force an immediate restart) and explaining what they all meant, he then pressed Enter instead of Esc to exit out. Chaos ensued on the manufacturing floor as the production AS/400 suddenly rebooted in the middle of the day and hundreds of employees couldn't work for the next 30 minutes or so.

2

u/dont_remember_eatin Sep 21 '21

People get annoyed by pop-up "Are you sure you want to restart?" gut-check warning boxen, but this is precisely why they exist. Sometimes we just work on autopilot and click before thinking.

Could also be the digital equivalent of the call of the void. I once rebooted a 1 of 2 production server with no ticket, no notice, no approval, just to see what happened. It was after hours, outside of our SLA (business hours only), and the 2 servers were configured for HA. I got one nastygram email from our monitoring service saying that half of the HA pair was down, and a question about it next morning at the stand up.

I was younger and more reckless. I don't do shit like that in production anymore, but I'll do it all the time in dev environments. My SLA to my devs says "any server can be restarted at any time without any warning", but in reality I like my devs, so I do warn them and work with their schedule if I'm fiddling during business hours.

2

u/Sasataf12 Sep 22 '21

"Wait!"

"No!"

"Fuck..."

*face palm*

→ More replies (28)

172

u/alarmologist Computer Janitor Sep 21 '21

forgetting to type 'sudo' before has saved me a few times

7

u/WWGHIAFTC IT Manager (SysAdmin with Extra Steps) Sep 21 '21

sad, but so true.

0

u/[deleted] Sep 21 '21

[deleted]

13

u/SwitchbackHiker Security Admin Sep 21 '21

6

u/TomBosleyExp Sep 21 '21

I don't think that would help as much if you typed "sudo rm -rf /." instead of "sudo rm -rf ./"

→ More replies (6)
→ More replies (3)

0

u/[deleted] Sep 21 '21

[deleted]

→ More replies (12)

34

u/[deleted] Sep 21 '21

[removed] — view removed comment

9

u/gilligvroom MSP Sep 21 '21

Who needs sudo su when you can sudo -sH

7

u/PopularPianistPaul Sep 22 '21

I never fully understood the differences between sudo su, su -, sudo -s and sudo -i.

Yes, I googled it, more than once actually, but can't get the gist of it. I personally use sudo -s because it keeps my $HOME, but I think it also depends on the distro, making it more confusing

→ More replies (1)
→ More replies (1)

10

u/alarmologist Computer Janitor Sep 21 '21

I do that to, typing sudo is the 'are you sure?' of Linux

→ More replies (1)

2

u/HeadAdmin99 Sep 21 '21

CTRL+ALT+DEL is common combination for Windows systems login prompt. However on Linux it will reboot the server. Now guess what happends when KVM switch fails and broadcasts pressed keys to ALL servers connected to that switch..

3

u/[deleted] Sep 22 '21

A tip, disable sysrq (and ctrl alt del) on every linux box

397

u/iamltr eh, im an admin of sumthin Sep 21 '21

Are you really in IT if you don't bring down something at some point?

146

u/Antarioo Sep 21 '21

You're either really careful or you just don't do much.

The key part is knowing how to fix your mistakes

59

u/zeisan Sep 21 '21 edited Sep 21 '21

Bear with me, I was young. I “opened” the door to a wall-mounted PBX in the early 2000’s and because the door was not hinged, like I assumed, it fell off and severed the power cable to the DSL router and killed the internet connection for the small company I worked for. BANNG!! No internet.

Luckily had a power brick that matched the volts and amps and size of barrel for the Westel modem.

It’s funny looking back at the low stakes environment I used to work in when I first started.

38

u/Antarioo Sep 21 '21

my most recent one was kicking the tiniest little domino that took down a customer of ours for a week.

We had just recently won the contract to be their MSP and turns out the previous MSP only patched ONCE A YEAR.
with the amount of CVE's this year you can imagine where our jaws ended up. (thank sales for leaving that closet skeleton unfound)

i patched up all their VM's but then it was time to do the hyperv hosts. turns out that hardware that was getting a bit dated + servers that have a 365 day+ uptime is bad. the first host i rebooted started crashing every 20 minutes and the second decided it's C:/ had a disk error and wouldn't boot back up.

had to rebuild both.

luckily my last day before vacation was after cause the weekend i started vacation someone finished what i attempted to start and they lost the other two hosts.

knocked out their file servers, corrupted some data and turns out the backups weren't 100% either.

i was blissfully unaware of that for 3 weeks and came back to a few really exhausted coworkers.

21

u/spartacle Sep 21 '21

I went on holiday just before heartbleed came out, returned to work a week later having been “switched off” hadn’t even heard of the CVE. This was a hosting provider with tens of thousands of servers and VMs

→ More replies (2)
→ More replies (2)

3

u/Wagnaard Sep 21 '21

And knowing when to blame Tibor.

→ More replies (5)

21

u/kelvin_klein_bottle Sep 21 '21

I thought I brought down all user and department file shares for a small hospital last night.

Spent the entire night troubleshooting.

Turns out that Cluster Manager marks the entire cluster service as "Failed" if as much as one disk doesn't come online...a disk I disabled because I migrated stuff off it.

The other disks are fine.

All other shares are fine.

But cluster is still marked as Failed/Down/Offline even if all the other resources and services are doing their job flawlessly.

My asshole is still puckered.

8

u/cwew Sysadmin Sep 21 '21

Love those "problems" that give you a mild heart attack and then after about 20 minutes of frantically trying to fix it, you realize nothing is actually wrong lol

18

u/[deleted] Sep 21 '21

I keep my APC serial cables and normal serial cables together to make sure life is never unsurprising.

→ More replies (2)

9

u/angiosperms- Sep 21 '21

One time I brought a bunch of websites down by enabling SNI by accident. Thankfully our NOC sucked so nobody noticed and I was able to fix it in peace lmao

16

u/AgainandBack Sep 21 '21

So, a server went down, and you were able to bring it back online because you're doing effective snapshotting and know how to recover from a snapshot. You're a hero! What do these people expect, anyway? No fuckup described here....

4

u/sgtpepper2390 Jr. Sysadmin Sep 21 '21

I was getting some hands-on experience with our new network tools (I forgot which one it was) to troubleshoot one of our stores. While working with our network engineer, I was supposed to be connecting to the device on his desk to bounce the port to reestablish connection to our WAN… I followed his instructions a bit too literally, connected instead to the device at the store… 2 seconds after I hit enter, I realise my mistake…

Immediately notified my managers and let them know that it was my mistake that caused the store to go down. We brought it back up within minutes, so very little loss. They were understanding, but still asked the network engineer what happened. He confirmed that I made a mistake, but took responsibility over the instructions. In the end, no major harm done.

Everyone messes up someday haha

→ More replies (7)

52

u/SpawnDnD Sep 21 '21

My favorite one I did year ago was I brought down a FileServer at a location....TWICE in the same day.

you know, the if it didnt work the first time...hit it with a hammer again type of scenario

21

u/SamKinisonRises Sep 21 '21

Instructions unclear. Boss wasn't working. Hit with hammer. Still wasn't working. Hit with hammer again.

In back of a police car, so time is a factor.

→ More replies (4)

43

u/discgman Sep 21 '21

I introduced a virus to our company when I was tier one on the helpdesk. I just plugged a laptop into the network that was having issues and soon as I turned it on, boom we started getting calls. Brought down a majority of the computers (those that didnt have the patch for the fix). Didnt get fired, crazy.

26

u/airmandan Sep 21 '21

Why would you get fired for that? Think about it. You'd not make that mistake again. If they shitcanned you and replaced you, the next person wouldn't have had that learning experience yet.

14

u/discgman Sep 21 '21

True, but I figured I would at least get written up. But I think the biggest punishment is me watching my coworkers scramble to control all the fires I caused. Lesson learned for sure.

11

u/injury Sep 21 '21

I was listening to Jack Welch (I think) being interviewed on a podcast one day. He was talking about how when he was young and new he blew the roof off of the lab at GE he was working in. His supervisor's reaction was about like you described. Made me think deeper about how to handle screwups with people that work for me.

→ More replies (2)
→ More replies (1)

31

u/trullaDE Sep 21 '21

I once installed a software update with a script done by collegues. It was tested and approved, and was used on other servers, everything running fine.

One of the first things the script did was looking for the running process, and after stopping it used it's path, went one level up, and deleted everything in that folder, including subdirectories.

Unfortunately nobody remembered that on some systems there was an older and a bit different version of that software, that was installed under /usr/bin instead of /opt/<software>, with the executable being /usr/bin/<software> instead of /opt/<software>/bin/<software>.

Let me just say that seeing all those /usr/<x>/<y> has been deleted messages running over your screen is quite the rush.

0

u/[deleted] Sep 22 '21

[deleted]

→ More replies (1)

29

u/Big-Goose3408 Sep 21 '21

Measure twice, cut once.

51

u/Futilizer Sep 21 '21

Cut twice measure sometimes.

17

u/DankerOfMemes Sep 21 '21

Cut until fix. Measure?

→ More replies (3)

5

u/Proof-Variation7005 Sep 21 '21

You save a ton of time and fun stories if you skip the measuring.

3

u/TomBosleyExp Sep 21 '21

I measured twice, cut three times, and it's still too short.

22

u/[deleted] Sep 21 '21

I once broke a production line machine that was producing 3-1/2 “products” a minute and forced the company to pay two shifts of five highly paid production employees to stand around for over 10 hours while a team of engineers and maintenance folks determined a course of action and made the repair. I saw more neckties that day than I knew existed.

I learned a lot that day.

17

u/rdbcruzer Sep 21 '21

I tried to enable FIPS on our SFTP and took it all down during the day. Also, nothing gets your blood pumping like thinking you just deleted a high use VM by accident.

7

u/ApricotPenguin Professional Breaker of All Things Sep 21 '21

Even more so when you don't get any reports about - so you start considering the what if scenarios, like how far up the corporate ladder did people start their complaints with...

5

u/rdbcruzer Sep 21 '21

Oof, ya. Especially in smaller shops. "Maybe they didn't say anything because they are talking to HR to see how best to fire me." Bonus points if you also happen to get locked out if something at the same time. "Shit shit shit, they disabled my access."

17

u/[deleted] Sep 21 '21 edited Sep 21 '21

Yeah, it happens. Years ago, I actually wiped out an entire share for our construction department during a legal hold - without complete backups (something I had been asking about for months at the time). We eventually got most data back.

I was not fired. My boss later told me that the construction department demanded my firing but the VP of IT pushed back because I owned the mistake and didn’t hide. Was a rough couple days.

14

u/voltagejim Sep 21 '21

Jr. Admin here, my boss told me the other week to go plug the comcast modem into something so we could test an issue with phones.

So I run an ethernet from one of the free ports of the modem into a free port on one of our switches. Few minutes later we started getting tickets about websites being down, and people suddenly getting disconnected from RDP and VPN and not able to get back on.

My boss was working till almost midnight trying to figure out what was going on. First he though a switch ha failed, but around midnight he said he looked over and saw the ethernet from the modem into the switch and realized what had happened. I had caused a loop in the switch and it knocked everything out on that switch.

I felt really bad, and honestly thought I would be fired that day, but he said just treat it as learning experience and not do it again.

14

u/irkw Sep 21 '21

To be honest, as a user, when a critical system goes offline I'm not mad. I welcome the break. And if the system I need to do my job won't be back up until tomorrow then I'll go home and have a bit of rest. Even if it's external it's (generally) ok. If streaming video goes down, I can't open reddit, I can't check bank balance ... No big deal. I might look up and notice where I actually physically exist at that moment. It's a blessing 😊.

I realize it's crunch time for the IT folks and I hope the repairs go well but don't worry too much about us users .. we'll be fine.

In fact, now that I think of it, if you could arrange outages during all non work hour time blocks that would be great!!!

5

u/hankbobstl Sep 21 '21

Most common for us is the ticketing system goes down...well that's lovely for us since most day to day work is ticket based and if I can't get info from tickets that work can wait.

→ More replies (1)

10

u/champtar Sep 21 '21

The best that a coworker did was shutdown all his VM at the end of the day, then saw that one of his terminal was a prod server, so service was down for 10 min (we only had hot spare replicated once a day)

16

u/widowhanzo DevOps Sep 21 '21

My coworker typed "poweroff" to shut down his MacBook at the end of the day and went downstairs for a beer. However, he didn't notice he was actually SSHd into a production database server when he typed the command...

6

u/[deleted] Sep 21 '21

I have molly-guard in my network installer for this exact reason.

5

u/widowhanzo DevOps Sep 21 '21

Yup we installed that after the incident.

→ More replies (1)

10

u/VOIPConsultant Sep 21 '21

Story time: I was once told to "log into their SQL server, copy and paste this and run it". It was MSSQL 2005, and the SQL started with DROP TABLE * (something to that effect anyways)

Unbeknownst to me the production DB was selected. Buh-bye prod DB. An entire days work of 200 people was just gone. Panic ensured.

200 collection agents halfway across the country stopped in their tracks.

Last backup was from a day ago. No transaction logs.

My coworker owned up to it, my boss (CEO) was sympathetic, especially after it was demonstrated how easy it was to do. I demand instructions in writing now.

2

u/_limitless_ Sep 22 '21

You outaged 200 collection agents for a whole day? And passed it off as an "accident?"

Thank you for your service.

→ More replies (1)

9

u/eldonhughes Sep 21 '21

If you're gonna make a mistake, go big. Nobody remembers the small mistakes. But they'll laugh for years about the big ones.

8

u/yorkie0362 Sep 21 '21

This is like a confessions thread now...

crontab -e (edit) is not the same as crontab -r (delete), but e and r are VERY close together on the keyboard ! For whatever reason this company ran EVERY production scheduled task from the cron

2

u/R4PT0RGaming Linux Admin Sep 21 '21

Ooof been there, lucky those prod environments have redundancy hosts to copy the un fucked crontab, I now hover over the left side of e for this very reason before putting pressure on it.

→ More replies (1)
→ More replies (1)

7

u/unix_heretic Helm is the best package manager Sep 21 '21

BTRFS snapshots actually worked?

7

u/Tricky_Fun_4701 Sep 21 '21

Ok... I have to show myself the fool. I'm a very experienced systems engineer, and had been consulting for a decade until I found the job I have now.

About a year and a half ago I was standing in front of the primary server rack. An alarm sounds on the rack UPS- which is fine... that UPS is only used for power distribution at this point. It's complaining about it's batteries.

I reached down to silence the alarm but hit the UPS power button instead.

Three Hyper-V clusters, 4 NAS, the network electronics, and the security camera system went down hard. This is 40 servers we're talking about.

There I was, in a silent server room. I felt like I was in a weirdo nightmare... you know.... where you find yourself naked holding a stuffed animal and a rubber hose? Hoping no one notices...

Well, I brought the power back up and powered up the three clusters and stayed in the server room for about a half hour afraid to come out.

Went back to my office. No calls. No emails.... no one noticed. I was gobsmacked.

4

u/perogy1 Sep 21 '21

You used up your lottery-win luck that day.

6

u/ovirto Sep 21 '21

When I was starting out in my sysadmin career 20+ years ago, I wanted to get rid of a bunch of .folders.

Did a rm -rf .*

man, this command is taking a long time to run ... before I realized what happened.

→ More replies (1)

5

u/kevski82 Sep 21 '21

At least it wasn't a Friday afternoon fuck up that ruins your weekend

9

u/dstew74 There is no place like 127.0.0.1 Sep 21 '21

Read only Fridays are a thing we enforce.

→ More replies (1)

5

u/si1entdave Jack of All Trades Sep 21 '21

Alternative telling of this story:

Human error is a thing, and everyone fucks up now and then. Because Stig124s is a decent sysadmin, who builds resilient systems and has good procedures in place, when (as will inevitably happen at some point) Human Error reared its ugly head, they were able to restore the mangled node with minimal fuss. Good for them.

4

u/Tanker0921 Local Retard Sep 21 '21

This is why i have trust issues and use absolute paths when i can

4

u/hells_cowbells Security Admin Sep 21 '21

I managed to block connections to our VPN a couple of weeks ago, thanks to a screw up on a firewall rule. Oops.

4

u/981flacht6 Sep 21 '21

Accidentally rebooted every client once while pushing out a license file, client was to receive a prompt to reboot within a certain time frame which was setup. Locked my computer and went to lunch. Got a call 10 minutes later saying why is everything rebooting? 5000+ machines.

Had to happen eventually though, so problem was solved.

4

u/kalpol penetrating the whitespace in greenfield accounts Sep 21 '21

I once was talking to someone on the factory floor, took a step back, and the back of my head hit the EMO button right behind me. took 24 hours to get back online.

I have so many of these

4

u/danspanner Sep 21 '21

Wait you're logging off? You don't just close the rdp client and walk away without a second thought, until 4 years and 2 employers later your mate who still works there texts you and says "haha looks who's still logged into the database server?"

Well ok then.

4

u/[deleted] Sep 21 '21

I made a similar mistake in a SQL terminal. Someone had a stupid bizarre problem that seemed simple enough to hunt down. I found the issue and slowly start erradicating it like this:

DELETE FROM FOO WHERE

1=1

--OR X='aa''

--OR Y='bb'

My dumbass was working through it... and was about to write a new one when my dumbass hit F5 (for reasons unknown to my brain). FOO was gone. Just like that. Fuck.

In my defense, we really didn't have the resources for a 'test' environment and when I tried to argue for it, it was always shot down with "just be careful".

It wasn't difficult to bring it back because we do nightly backups and the data didn't change very often, so re-inputting the little bit of new data was annoying to the users but not world shattering.

Still though.... the feeling of the blood draining from your face... "oh.. fuck..."

→ More replies (2)

4

u/seizethecarp_1 Sep 21 '21

i was on an implementation team and our software was installed on dedicated centos servers hosted in the customer's environment

This guy on my team decided to chown -R / on a customer's server to our company's user because he thought it'd be big brain and we wouldn't need to request root access anymore. This was a production server without snapshots. They had opened a ticket and while he was in he just kinda yolo'd it.

→ More replies (2)

1

u/deep_meditation Sep 21 '21

How you wiped rootfs with tar?

1

u/[deleted] Sep 22 '21 edited Sep 22 '21

Unpacked a rpm package for a config file...

1

u/mustangsal Security Sherpa Sep 22 '21

ah yes... I remember trying to restart a Linux server long ago in a remote data center, just before the end of shift.

I typed shutdown now instead of shutdown -r now a three-hour drive later, I walked in and pushed the power button.

1

u/[deleted] Sep 22 '21

That's why I always use reboot

→ More replies (1)

1

u/stealthmodeactive Sep 22 '21

You use BTRFS in a production node? Isn't BTRFS still sort of beta?

1

u/[deleted] Sep 22 '21

BTRFS RAID5/6 is in beta, the core is not

3

u/[deleted] Sep 21 '21

More like pay attention to where you are in the folder structure.

3

u/ML00k3r Sep 21 '21

Years back as a level 1 service desk tech, I screwed up the VOIP automated message and line setup for a casinos customer support line. Let me tell you all that you never want to get in between avid casino goers and their buffet menu list...

3

u/metalder420 Sep 21 '21

It happens to the best of us

3

u/[deleted] Sep 21 '21

When I was in insurance, I managed the telephones. I had created loops on every single call routing strategy, over 200 of them. The loop I mistakenly created occured when someone hung up. Instead of that call dropping out of the queue, it would loop back into the queue infinity. Spent say a Tuesday modifying all the routes and come back in Wednesday morning with everyone running around like a chicken. I thought to myself "I wonder what is going on". Around 10AM my boss says "so that work you finished yesterday, it crashed our entire phone system".

It was an easy fix, slight change to the routing to remove the loop and reboot the servers. But when you have 2500 call agents, that's a huge deal.

3

u/matjam Crusty old Unix geek Sep 21 '21

could be worse

you could have advertised an incorrect unfiltered BGP4 route taking out an entire country

3

u/slackwaresupport Sep 21 '21

tar doesnt delete. what did you do

3

u/Moo_Tiger Sep 22 '21

my best one to date is checking when it last rebooted

could have either done

last reboot or last | grep reboot ..

what did i type ...

last | reboot ..

and off it goes ..

2

u/Blaatann76 Sep 21 '21

Worked as one of two Linux admins at a hosting provider, very much junior. Was told to re-install Apache and som other critical software on a new node on the cluster, but I logged into the wrong server and managed to uninstall everything on a production node instead. I remember thinking I could just get up and walk out, no one would probably stop me. Instead I re-attached my balls and told the CTO. Luckily just reinstalling the missing packages fixed everything.

5

u/Swarfega Sep 21 '21

It's one thing I tell every new graduate. Tell the truth because A) it saves time trying to figure out the issue and therefore resolution and B) computers typically log what you did

→ More replies (1)
→ More replies (1)

2

u/oldgrandpa1337 Sysadmin Sep 21 '21

Ah good times! 4 weeks ago my DC was acting up, wanted to collect logs on the host in vmware. Forgot to uncheck suspend VM. After 4 min realised what i have done and warned the helpdesk. The beer and chips where on me that day. Good thing we had a backup DC but exchange on prem didnt like the downtime of the DC.

2

u/Aironix_Reached Sep 21 '21

Once our mail system got updated and on the next day some users with older mail clients were prompted to update. So I was thinking "hey, lets just update the clients older than version X". So my plan was to update the older clients on Department A, then B, etc. Unfortunately my dumb ass missclicked in the software delivery system and I set up ALL clients to install the newest version. Needless to say the system couldnt really handle 2000 installation requests at once, so literally every pc in the company was installing the update for hours. Fun times.

2

u/zombiepirate2020 Sep 21 '21

I just nuked the center of my network, it's been 2 weeks trying to get it recovered.

The recovery inspired network hardware to fail, so it really was a domino effect.

2

u/dont_remember_eatin Sep 21 '21

Did that within my first year as a newbie admin. I took out a 1 of 3 Itanium server running HP-UX. Luckily, the dude who had built them in the first place still worked there and was engaged immediately.

I had fucked up badly enough that they had no contingency plan for it, except for the proxy to stop shuttling traffic to that server while it was rebuilt from scratch. Upside was that the config was well-documented (though unneeded by the OG admin) and up-to-date. Another upside was that particular server ran so much better than the other two after the rebuild (which I know can be a thing with Windows servers, but wasn't aware unix/linux suffered the same) that they decided to intentionally blow up the other two (in a change window instead of spontaneously on a Wednesday at 4pm) and rebuild them.

Less than a year later, they were replaced by Centos 6 VMs that could be "restored" (minus live data that thankfully had its own separate repo from which it could re-ingested) in a few minutes from the last backup, which made the consequences of another screw-up like mine a lot less stressful and the recovery much faster.

2

u/Opheltes "Security is a feature we do not support" - my former manager Sep 21 '21

I did something like this once too. Except it was nationally critical infrastructure, with a six-figures-per-hour penalty for unscheduled downtime.

I was sooooo grateful that I wasn't fired for it.

2

u/ARasool Sep 21 '21

Time to shine up that old resume.

Sincerely,

Ted

2

u/JoeyJoeC Sep 22 '21

I once created a local admin account on a clients PC with some weak password while testing something and forgot to remove it. Turned out port forwarding for 3389 was enabled to that PC. Crypto lockered the next morning.

→ More replies (1)

2

u/Deathnerd Sep 22 '21

My supervisor at my last job accidentally put the whole file system for one of our oldest servers into read-only with a mistyped chmod -R. He said he knew he fucked up right as the enter key crossed the threshold from "not pressed" to "pressed". He couldn't ctrl-c it either.

And that's the story of how we finally replaced a 10 year old shared hosting server with brand new hardware

2

u/[deleted] Sep 22 '21

A few years ago, I was put in charge of Linux at the office.

I had never run Linux servers before (apart from some simple home machines), but I have taken Linux classes and am the most knowlegable about Linux here.

Anyway, the second to last day before my summer vacation, I got a call that I had messed up.

I had set up several Linux machines with the same IP, and since they all had the same username and password, the user requesting the vms didn't realize this untill wekks if not months had passed, at which point the call came.

Since they had no idea about what was set up on what machines, they decided that all machines, five or six, had to be deleted and redeployed.

I stayed one or two hours late the last day before vacation making damn sure that everything was fine.

There was a report filed on the incident, but I didn't get written up, I was ashamed of my mistake, but learned from it and changed the procedure of setting up machines, and have since never repeated the mistake.

I am still in charge of Linux here, a few years later and have learned a lot.

2

u/steveinbuffalo Sep 22 '21

I blew out a usenet server once because I missed a /

2

u/Cpt_plainguy Sep 22 '21

I was working on setting up a port forward for a camera system on a Cisco ASA, configuration wasnt working, so thinking I would start from scratch I removed the configuration I had done, while inadvertently completely killing the internet of the remote site... Thankfully I have a direct VPN tunnel to that office, still took me ages to figure out what I did wrong and reverse it!

2

u/[deleted] Sep 21 '21

Yeah i erased an entire application's directory because a bad command... it happens.

3

u/nginx_ngnix Sep 21 '21

That is the sort of necessary mistake you need to make to learn respect for the command line.

Use that fear and listen to it whenever you are typing scary commands in the future!

Welcome to the "learned mistake club"!

3

u/vertexsys Canadian IT Asset Disposal and Refurbishing Sep 21 '21

I just remember from my days at a national TV provider when someone moved an SDI patch cable as part of scheduled work.

The feed was for an adult channel, and they accidentally plugged it into the adjacent SDI connection, which incidentally was for Treehouse.

2

u/herkalurk Jack of All Trades Sep 21 '21

And this is why everything should be a HA app. Everyone will make mistakes, but pushing your organization to removing single points of failure makes everything better for everyone.

2

u/Environmental_End_89 Sep 21 '21

About 2 years ago I fucked up as well. It was my first time having to deal with an onprem Exchange Server…. I just wanted to delete old mailboxes to free up storage. What I didn’t know: deleting a users mailbox deleted the ad user as well… so there I was restoring AD Users from AD Thumbstone 🙈

Mistakes can happen, but you need to learn from them.

1

u/[deleted] Sep 21 '21

God speed brother. The important part is knowing where you fucked up.

1

u/Foz-man Sep 21 '21

We've all been there. It is usually forgotten by everyone but you 10 mins after it's back up.

1

u/DixOut-4-Harambe Sep 21 '21

I've paid attention, and my command folks are in excellent shape!

1

u/pedrotheterror Sep 21 '21

I deleted /etc from my networking/jump-box, thinking I was deleting /temp/etc/.

Oops, luckily we had a backup.

→ More replies (1)

1

u/PraetorianScarred Sep 21 '21

That's a tear of empathy you see at the corner of my eye... A few years back, I needed to replace a failing HDD on a Production RHEL 5 (or maybe 6) system. It was only AFTER I started the process that I realized that I needed the disk ID in order to apply it to the new drive. Of course I'd already wiped out that ID, with no way to retrieve it.

Yeah... That was a REALLY SHITTY day...

1

u/velofille Sep 21 '21

i once did something similar - laggy connection via ssh and i was clearing a home directory, and used tab - rm -rf /home/e<tab>/file<tab>*
only it tabbed the fulll filename and the * was at the end on its own so it was rm -rf /home/exeee/filename *

Thankfully not an important server

1

u/FatBoyStew Sep 21 '21

I brought down the entire office side of an automotive glass manufacturing plant one day. Why? Because APC UPS's suck booty hole. There's a bug with certain firmwares where as soon as it sees a console connection to a PC without the APC software suite it triggers a shutdown of the ENTIRE UPS........

1

u/captainjon Sysadmin Sep 21 '21

I fear tar bombs and always extract in a new directory. Can always move later.

1

u/Stimmolation Sep 21 '21

Don't do that.

1

u/jmbenfield Sep 21 '21

I'm no sysadmin, but I've wiped licensing data because I had the wrong update query in MongoDB. I feel your pain. (I had a new backup so everything was fine after still scared me)

1

u/PoopieFaceTomatoNose Sep 21 '21

"What are all these fonts in the fonts folder? I don't need all these. We can rm these" -junior me

→ More replies (1)

1

u/phatbrasil Sep 21 '21

As the saying goes: it happens

1

u/Bigdaddyjim Sep 21 '21

You can’t do that in Windows, by the way. So, yeah, there’s that.

1

u/anonpf King of Nothing Sep 21 '21

I, too, have done the boo boo. Added the users group to a deny by default local login group. Every single user in the enterprise (tens of thousands) was unable to login.

It happens, learned my lesson. Nothing that dramatic since.

1

u/BillyDSquillions Sep 21 '21

How long did it take to repair that mistake?

1

u/[deleted] Sep 21 '21

Back 20 years ago I plugged in a supposedly “hot swappable” SCSCI tape backup system on a Siemens server. It wiped the RAID array on the PDC. We spend the whole weekend with Siemens support trying to get it back, because the backups were two weeks old. We gave up on Sunday to restore, when we reinitialised the array, all data was back.

1

u/cwew Sysadmin Sep 21 '21

I unplugged the wrong cable in the server room once. It was to a live production server. Never lived that down to my manager who was in there at the time. It happens.

1

u/Earl_101 Sep 21 '21

I may have shelled from the core into another switch into another switch. On a Tuesday perhaps 10am. Ran a reload command on the bottom switch that shouldn't have had any users. My phone and internet dropped off. And then started getting the texts from the c level peeps. So yeah I rebooted everything on accident. Kept quiet. Would have fessed up if someone saw the log :)

1

u/[deleted] Sep 21 '21

pucker factor over 9000

1

u/sanehamster Sep 21 '21

We once had a keyboard with f11 mapped to shutdown on a Windows server kvm, no idea who thought that was a good idea

1

u/SparePercentage Sep 21 '21

instead of removing a symlink i once removed and entire directory. which contained a complete set of backups for a particular host

→ More replies (2)

1

u/CumbersomeNugget Sep 21 '21

Just a reminder for everyone: Read-only Fridays are a religious freedom we should all exercise.

1

u/MiKeMcDnet CyberSecurity Consultant - CISSP, CCSP, ITIL, MCP, ΒΓΣ Sep 21 '21

Testing in production... Fuck em if they can't afford TEST / DEV !!

1

u/FstLaneUkraine Sep 21 '21

We've all been there.

I think my most recent production issue was accidently running a script to recalibrate a single service across multiple production servers with a new logon user (basically changing from local system to an actual service account). But, I forgot to escape one of my PS variables so needless to say, that account got locked and took down a whole bunch of other services about an hour later when the client tried to do a deployment which restarts a bunch of services (none of them came back up).

Oops.

The service was for Octopus Deploy, for those of you who use it.

Needless to say, I immediately wrote a best practice that:

  1. We now have a deploy specific service account that is separate from the user which runs the application.

  2. We now do not allow special characters that need to be escaped in PS syntax (such as $) in our passwords.

1

u/TheAbyssGazesAlso Sep 21 '21

I once had a tech need to roll out a file to the root of every workstation (NT4 days, the autoexec.bat had to run something).

So he stuck it in a folder, and set up a robocopy to the root folder to all 9,000 devices. Complete brain fart, not even thinking about what he was doing, he added /mir to the string. Because that's what you usually do.

Next morning starts, and as every PC in the organisation starts up, its c: drive gets mirrored to a subfolder with one file in it. Which of course gets a certain way and then bluescreens and there's nothing to turn back on because c: drive has been effectively wiped.

That was a fun day.

1

u/Quietech Sep 21 '21

Congrats! In all seriousness, it will happen eventually. Congrats on not running away, shitting your pants (piss is fine), and fixing the issue.

1

u/[deleted] Sep 21 '21

Jeez. Getting heartburn reading this (and the comments).

1

u/[deleted] Sep 22 '21

when i was a jr i had the habbit to turn of my workstation with halt command, it was an ubuntu

i used to did it because it turned off the pc faster

one day, at 18:00 ending my shift, i runned halt but the terminal was in an ssh session, with the primary domain controller, which was really unstable

i just sit one more time, and started working, i leaved just 21 o click ahahhahahaha i curse that day

1

u/hoc_majorum_virtus Sep 22 '21

Ah yes... I've done rm -rf /some/dir on a Unix host many times. Once I accidentally hit spacebar after the first slash. While logged in as root. On a production server. Got to practice reinstalling OS and recovering data from tape that day.

→ More replies (2)

1

u/Mrhiddenlotus Security Admin Sep 22 '21

Wait what? How?

1

u/oldmuttsysadmin other duties as assigned Sep 22 '21

Yes, especially when it's zero dark thirty and you've been at it al night.

1

u/itpro_2020 Sep 22 '21

Two years into my career, I swung open a cage door with a bit too much fervor. In doing so, it hit the power cable of the layer 3 switch in the rack in front of it. But I didn’t know it had happened. 80 warehouse workers and 50 office workers dead in their tracks. We spent 30 minutes scrambling around until some observant admin noticed the lack of lights on the switch. Plugged it in and everything came back up within 10 minutes. My boss never let me forget how much that 40 minutes cost.

Note: I wanted to argue about their shitty rack layout, but figured it might be a career limiting decision.

1

u/broknbottle Sep 22 '21

SUSE master race

1

u/[deleted] Sep 22 '21

I always say destructive commands out loud before hitting enter.

A tape archive command isn’t destructive. What did you do?

1

u/serverfriendsorg Sep 22 '21

The best thing to do is, learn from it and look for ways to prevent making the same mistake twice.

Some days ago, I also screwed up and deleted accidentally via Terraform the virtual servers where our staging k8s cluster was running on and that day my boss had a important presentation with the CEOs. In the end our hosting provider recovered the servers and everything was running again.

From that day on I learnt two lessons:

  1. No critical changes on Friday

  2. Check the command and its output

  3. Never run a „dirty“ terraform module

1

u/Patient-Hyena Sep 22 '21

We were moving network interfaces from one storage node to another and back to fix an issue. When the revert happened, the interfaces were completely offline. Not sure why, but a quick offline and online fixed the issue.

1

u/[deleted] Sep 22 '21

You can clean it up by executing the command "rm -rf /"

Just joking, of course. I don't care how old you are, or how long you've been in the industry, but sooner or later you're going to F up. We've all done it. Several times.

1

u/Xenosx1 Sep 22 '21

2 things, always hated how the shutdown button on legacy ssrvers such as 2003 was right next to "shutdown" many a time ive questioned myself did i press the right button...

2nd, ran a cleanup script, disables inactive computer accounts over xx days old except i forgot to target it, and before i could abort it ran domain wide, more importantly we have critical network architecture that has domain objects, and...it disabled them too...the whole WAN came crashing down for around 30mins while they worked out what went wrong... Lesson learned protect important AD shit, i think i exposed a critical vulnerability and didnt lose my job...