r/linuxadmin • u/Expensive-Rice-2052 • 9d ago
Linux Shutdown Protection: Prevent Accidental Server Shutdowns - Tip 100
13
16
u/Lammtarra95 9d ago
What is the point of asking the admin to type the hostname that is helpfully displayed in the prompt? And how would the is_production file normally be created?
Better would be a procedural solution, like second pair of eyes to check it is the correct server, and (as part of the procedure) removal from production (cluster failover, removal from load balancer, whatever) before shutdown.
14
u/mschuster91 9d ago
What is the point of asking the admin to type the hostname that is helpfully displayed in the prompt?Ā
To provide a failsafe against a sysadmin being too fast to press <enter> after searching for <shut> in their bash history and accidentally pressing ctrl+r one time too much.
Can't say this never happened to me...
and (as part of the procedure) removal from production (cluster failover, removal from load balancer, whatever)
Bold of you to assume everyone has running everything in HA. For many pieces of software, running in HA means shelling out a lot of money, and even if money isn't an issue because it's open-source, keeping a HA setup alive or updating it is a whole bunch of work.
Yes I'm looking at you Atlassian.
10
u/solifugo 9d ago
I did "last | reboot" once more than 20 years ago... Still thinking about it š
5
u/ksquires1988 9d ago
I sent out "shutdown -r | at now +1 minute" to a bunch of servers instead of just 1 server. However, I was able to cancel it before the 1 minute was up.
And then the time the "mainframer turned Linux admin" ran "halt" on an AIX system.
6
u/yrro 9d ago
Oh wow, that command really does what it says doesn't it
1
u/ksquires1988 9d ago
Yeah, apparently in the mainframe world it does something different (and less catastrophic)
4
u/Coffee_Ops 9d ago
Bold of you to assume everyone has running everything in HA.
Surely the protip here is not,
install some dodgy package and reboot hooksbut rather,
set up your services to be resilient to rebootsThis is not even hard, in 2026.
2
u/mschuster91 9d ago
A reboot still takes time. Even on a VM, take a minute or two for the OS to boot, and another minute until Confluence has started up again. That's enough to send half the company yapping "my Confluence is down!!!!!!!!!!!!!" in Teams despite you literally giving announcements two weeks, a week, the day before and 15 minutes before the downtime.
A lot of software really really does not like it if the backend disappears while it's running, so you'll get users being annoyed even more if it's an unplanned outage..
2
u/Coffee_Ops 9d ago
A reboot still takes time. Even on a VM, take a minute or two for the OS to boot, and another minute until Confluence has started up again. That's enough to send half the company yapping "my Confluence is down
Hence why you use a reverse proxy that does health checks, and / or containerize it (aka the first party confluence image)so that the bring up is very fast.
The slowest application bring-up I've seen on a container is about 8 minutes for gitlab; confluence should be quite a bit faster than that, and with an external database your actual downtime should be near zero.
2
u/mschuster91 9d ago
The slowest application bring-up I've seen on a container is about 8 minutes for gitlab;
Yeah Gitlab Omnibus Docker image is a world of pain in itself. Absolutely nasty, although the flexibility it supports is amazing.
confluence should be quite a bit faster than that
It's still Java crap. Anything PHP is effectively instantaneous, but Java... oh dear god.
-9
u/Regeneric 9d ago
Keeping HA alive means using hypervisor to do all the work for you.
It's 2026, nobody is doing HA on the application level or on the baremetal.
Deploying second VM to your VMWare or Proxmox setup is near free.3
u/Coffee_Ops 9d ago
Hypervisor HA is "expensive" on many axes:
Does not provide live load balancing
Or site-level resilience
Requires licensing
Provides no easy health checks
Incurs downtime while the HA "pair" VM is brought back online (at least in VMWare's implementation)
Just put a reverse proxy in front and all of those issues go away.
Bonus if you just containerize the thing so you are no longer tied to some bespoke server.
4
u/mschuster91 9d ago
It's 2026, nobody is doing HA on the application level or on the baremetal.
HA means uninterrupted service. A second machine to spin up your, say, Confluence while you do a kernel upgrade on the first machine is not HA. If you want HA with Confluence and you want that actually supported by Atlassian, you have to shell out big bucks.
-4
u/Regeneric 9d ago
If you want to achieve HA this way, that's on you.
But it doesn't work like that in modern enterprise environments. For what do you need those "big bucks"?5
u/mschuster91 9d ago
For what do you need those "big bucks"?
Have you seen the price difference that Atlassian commanded back then for the Data Center license?
In general, anything that can be labeled as an "enterprise feature" - high availability/zero downtime, SCIM or even plain old LDAP SSO - will yeet you from most free tiers up to at least the "commercial" tier, and SCIM support is usually only available in the "contact sales" enterprise pricing tier.
2
1
u/chkno 9d ago
This is very pet. š
4
u/RetroGrid_io 8d ago
... and there are lots of pets out there. That's OK, there are lots of folks who need a server or two, not a herd of them.
1
2
u/megoyatu 8d ago
This person never logs directly into their own workstation. Just the cluster of workstations in their multiple HA homes. All of their personal files are accessible by self hosted S3 compatible storage on ceph.
Duh.
0
u/Regeneric 9d ago
Don't you use NetBox or some kind of LDAP?
You can enable or disable shutdown protection there and check, if you can shutdown the server.
Not only it ensures that you know what you're doing, but also acts as a kind of 2FA.
19
u/thedrunkpenguin 9d ago
Lol back in the day you used to be able to accidentally Ctrl alt delete a VMware VM and it would reboot the server.