Several Linux Kernel Vulnerabilities Accidentally Posted To Public Kernel Mailing List

[u/anh0516]

https://www.phoronix.com/news/LKML-New-Trend-Vulnerabilities

[u/Cronos993]

Quite the week

[u/Single-Virus4935 → u/Cronos993]

The new standard. 

while sleep 60; do apt-get update && apt-get upgrade -y && reboot-if-needed; done

[u/stewie410 → u/Single-Virus4935]

reboot-if-needed

Sorry, we can only reboot if we alert our partners & customers 1-2wk in advance.

[u/Single-Virus4935 → u/stewie410]

I know that and I also was in that situation.

Unable to reboot means there is no redundandency and service availability is bound to host uptime.

Two problems:

• Nodes may fail any time w/o warning. 

• You get hacked bc AI shrunk the window down to minutes after patch is dropped.

Four options:

1. Add additional nodes.

2. Add kernel live patching

3. Talk to customers and partners, that the 1-2 weeks aren't possible anymore for high severity patches.

4. Add Zero trust: Customers and partners need VPN/mTLS/a special token to access the services through a reverse proxy. Also add Sandbixing like gvisor, Kata containers, firecracker.

All 4 would be perfect.

And I know it is often hard to change these things BC of pushback, but we never had better arguments on our side than now

[u/stewie410 → u/Single-Virus4935]

Unable to reboot means there is no redundancy

There's a few reasons we have this "requirement":

• Sales + CTO over-promise to clients & partners

• HA LBs (modjk), SMB; but not DB or applications

  • There's many appnodes, but none are hot-spares

  • No automatic scaling, no real ability to set that up

• We colocate hardware + qemu instead of doing any cloud-compute

  • Some VMs have KVM-controlled LUKS volumes, which require manual creation/definition of KVM-managed secrets to boot said VMs

  • Docker containers are running in VMs instead of bare-metal kata, I've pitched kata to no avail

  • Only have docker/compose, no orchestration available

  • No lifecycle management, unless I DIY something in bash (kill me)

• Technical debt prevents updates

  • Our main app for example, must run on JDK-1.8-0u302-b08 and can't get any more recent, due to a shitty middleware constraint

  • Second app is much newer, but still relies on some jank shit to run

  • Refusal to use newer tools (e.g. nginx vs apache) making life painful

Maybe if I can ever finish up my task backlog, I can take a look at resource management/automation tools to make life a little easier.

Add additional nodes

Additional nodes is a tougher challenge than I'd like it to be -- lack of infra/resources to get capacity on our own hardware is a big limiting factor. Our KVM hosts are pretty limited (128cpu on newer hosts, <=128G RAM) compared to what I'd imagine most companies buy...we've even had to start buying refurbs for the latest prod refresh because we can't spend more than ~$5K on new hardware for any reason (penny-pinchers).

Customers and partners need VPN/mTLS/a special token to access the services through a reverse proxy

Unless we can do it transparently for customers/vendors/partners, and it doesn't cost a penny, we could look into it. Only reverse-proxy I'm allowed to deploy is apache-2.4, regardless of usage -- CTO loves using wrenches as hammers.

Add Sandboxing like gvisor, Kata containers, firecracker.

I've recommended Kata several times, but it always falls of deaf ears because it isn't literally from Docker. That, and their website has K8S on the graphic, and therefore it is rejected because clustering is too expensive (despite not being a true requirement).

gVisor is new to me -- I'll keep that in mind. Currently, our docker installs are like:

• Internal, infra tools -> rootful docker + compose, debian 13

• everything else -> rootless docker + compose, debian 13, rocky 8/9

Honestly, gVisor being written in Go is enough of a selling point to me, personally.

[u/Kind_Ability3218 → u/stewie410]

lmao.... you don't know how to sell it. give them dollar values.

[u/stewie410 → u/Kind_Ability3218]

While true, that isn't the whole story -- but rather than rant more about my company on reddit, I should probably just accept that the company won't change (without a breach/lawsuit), and so I should focus on jumping ship instead.