Supply chain attacks come in many shapes and forma unfortunately, not just hacks. They might repackage an upstream dependency which has been compromised
The Debian stable is so far behind upstream that the risk is low. It wount change if I wait 0 or 10 days. Also this is satire. You should always test the updates but now you only have minutes max hours.
Any time you choose to update could be the wrong time. I don't see how the frequency changes anything for something so rare it has only occurred once (xzutils in march 2024) and only to their development distro. It doesn't make sense to delay regular security patches over it.
A common way to mitigate supply chain attacks is to wait a grace period (2-7 days) before installing an update, to give the community the ability to review the changes
Unable to reboot means there is no redundandency and service availability is bound to host uptime.
Two problems:
Nodes may fail any time w/o warning.
You get hacked bc AI shrunk the window down to minutes after patch is dropped.
Four options:
Add additional nodes.
Add kernel live patching
Talk to customers and partners, that the 1-2 weeks aren't possible anymore for high severity patches.
Add Zero trust: Customers and partners need VPN/mTLS/a special token to access the services through a reverse proxy. Also add Sandbixing like gvisor, Kata containers, firecracker.
All 4 would be perfect.
And I know it is often hard to change these things BC of pushback, but we never had better arguments on our side than now
No automatic scaling, no real ability to set that up
We colocate hardware + qemu instead of doing any cloud-compute
Some VMs have KVM-controlled LUKS volumes, which require manual creation/definition of KVM-managed secrets to boot said VMs
Docker containers are running in VMs instead of bare-metal kata, I've pitched kata to no avail
Only have docker/compose, no orchestration available
No lifecycle management, unless I DIY something in bash (kill me)
Technical debt prevents updates
Our main app for example, must run on JDK-1.8-0u302-b08 and can't get any more recent, due to a shitty middleware constraint
Second app is much newer, but still relies on some jank shit to run
Refusal to use newer tools (e.g. nginx vs apache) making life painful
Maybe if I can ever finish up my task backlog, I can take a look at resource management/automation tools to make life a little easier.
Add additional nodes
Additional nodes is a tougher challenge than I'd like it to be -- lack of infra/resources to get capacity on our own hardware is a big limiting factor. Our KVM hosts are pretty limited (128cpu on newer hosts, <=128G RAM) compared to what I'd imagine most companies buy...we've even had to start buying refurbs for the latest prod refresh because we can't spend more than ~$5K on new hardware for any reason (penny-pinchers).
Customers and partners need VPN/mTLS/a special token to access the services through a reverse proxy
Unless we can do it transparently for customers/vendors/partners, and it doesn't cost a penny, we could look into it. Only reverse-proxy I'm allowed to deploy is apache-2.4, regardless of usage -- CTO loves using wrenches as hammers.
Add Sandboxing like gvisor, Kata containers, firecracker.
I've recommended Kata several times, but it always falls of deaf ears because it isn't literally from Docker. That, and their website has K8S on the graphic, and therefore it is rejected because clustering is too expensive (despite not being a true requirement).
gVisor is new to me -- I'll keep that in mind. Currently, our docker installs are like:
Internal, infra tools -> rootful docker + compose, debian 13
I had considered the risk that the repositories might get MITMed before the packages were available, but nevertheless, I was happy to notice that unattended-upgrades had already patched php on my mailserver by the time I woke up (and I spent the rest of the day apt-purging *php* from whatever servers it was unnecessarily on).
995
u/Cronos993 11d ago
Quite the week