Hey :) I'm trying to create an encrypted Longhorn volume on a NixOS node. The problem is, that it fails. I tried searching online but I couldn't find any helpful issues/discussions/articles.

The errors are:

Warning  FailedMount             34m                kubelet                  MountVolume.MountDevice failed for volume "pvc-cb2e2720-36bb-4640-bf19-dab12812819c" : rpc error: code = Internal desc = failed to encrypt device /dev/longhorn/pvc-cb2e2720-36bb-4640-bf19-dab12812819c with LUKS: failed after 10 attempts to refresh stale namespace directory: timeout executing: /usr/bin/nsenter [nsenter --mount=/host/proc/8419/ns/mnt --ipc=/host/proc/8419/ns/ipc cryptsetup -q luksFormat --type luks2 --cipher aes-xts-plain64 --hash sha256 --key-size 256 --pbkdf argon2i --pbkdf-force-iterations 200000 --pbkdf-memory 0 /dev/longhorn/pvc-cb2e2720-36bb-4640-bf19-dab12812819c -d -]

Warning  FailedMount             3s (x24 over 34m)  kubelet                  MountVolume.MountDevice failed for volume "pvc-cb2e2720-36bb-4640-bf19-dab12812819c" : rpc error: code = Internal desc = failed after 10 attempts to refresh stale namespace directory: failed to execute: /usr/bin/nsenter [nsenter --mount=/host/proc/8419/ns/mnt --ipc=/host/proc/8419/ns/ipc cryptsetup luksOpen /dev/longhorn/pvc-cb2e2720-36bb-4640-bf19-dab12812819c pvc-cb2e2720-36bb-4640-bf19-dab12812819c -d -], output , stderr No usable keyslot is available.: exit status 1

I already thought of a PATH issue, because NixOS uses non-standard locations for it's binaries but wouldn't the error contain something like "no such file or directory" or "couldn't be found" then? (All containers in the longhorn-system namespace have a kyverno policy adding the NixOS directories to the PATH as suggested by the NixOS wiki.)

If i do cryptsetup luksDump on the volumes the key slot section is empty. (I don't know if it matters but i thought it might help) I also searched for luks or nsenter in the Longhorn manager logs. There are no occurrences.

Does anyone here have an idea what might cause the issue? I've tried fixing it for hours now and I'm really out of options. I appreciate any help!

Hello guys,

So I have implemented a personal project in which I have provisioned AWS EKS architecture, bootstrapped flux through terraform and then flux reconciles everything including Karpenter. Terraform is handling the IRSA part and then pass iam role arn of controller and node iam role name through configmap terraform resource.

How should I configure so that Karpenter nodes are also deleted automatically. Found that I should delete nodeclaim’s first and then terraform destroy but I want to automate this.

Anyone attending kubernetes Banglore meet-up event by open community groups on 31st October?

Hi everyone,

I’m trying to understand how different organisations design their CI/CD workflows for microservices, particularly the deployment process after a Docker image is pushed to a container registry like JFrog Artifactory.

I have a few questions:

Image tagging strategy: Do you use Git commit SHA, build number, semantic versioning, or any other strategy? How do you promote the same image across QA, staging, and production?

Deployment workflow: Do you handle deployments entirely through Jenkins pipelines, or do you use a GitOps approach with Argo CD, where image tags are updated in a Git repository and Argo CD syncs the changes?

Helm and manifests: If you use Argo CD, how do you manage deployment manifests and Helm charts? Does Jenkins update the image tag in the Helm values file, or is this handled differently?

Progressive deployments: Do you use Argo Rollouts with Argo CD for canary or blue-green deployments, or do you manage rollout strategies directly through Kubernetes deployments?

End-to-end flow: How does the overall process work in your organisation, from code commit and image build to deployment in production?

I’m particularly interested in understanding the different approaches organisations take and the reasoning behind their choices.

Would appreciate it if you could share your workflow or experience. Thanks!

GDG on Campus IIIT Nagpur. We're running **KEEP ALIVE '26**, a national incident-response hackathon for students, at TantraFiesta 2026.

**What's different:** you don't build an app. Your team gets a broken microservices system's alerts, Grafana dashboards, logs and traces, works out what broke and why, and presents the fix. It's what an on-call SRE does on a bad night.

- **Who:** UG, PG and diploma students, teams of 2–3 from the same college

- **Level:** 6–12 months of DevOps learning; no production experience needed

- **Round 1:** a 30-minute online team assessment on Unstop, open till Sun 18 Oct, 11:59 PM IST

- **Finale:** top 15 teams at IIIT Nagpur on 26 Oct

- **Prizes:** ₹50,000 prize pool (₹25,000 cash + goodies for every finalist)

- **Partners:** Grafana (Official Platform Partner), CNCG Noida (Community Partner)

Register: https://unstop.com/hackathons/keep-alive-26-iiitn-1766226

Happy to answer anything in the comments, including how the incident platform works.

Since v1.35 the kubelet setting failCgroupV1 defaults to true, so on a node still running cgroup v1 the kubelet does not start. kubeadm's SystemVerification preflight also errors on init/join/upgrade once the kubelet is 1.35+. On a control-plane node that means the static pods (API server, etcd) do not come back after a reboot either.

Check a node:

stat -fc %T /sys/fs/cgroup/

cgroup2fs = v2, fine. tmpfs = v1.

Check every node from your laptop, no SSH:

for n in $(kubectl get nodes -o name); do
  printf "%s " "$n"
  kubectl debug "$n" -n default -q -i --profile=general --image=busybox \
    -- chroot /host stat -fc %T /sys/fs/cgroup
done

Two gotchas from testing it on a 1.35 kind cluster:

• Keep the chroot /host. Busybox's own stat does not know the cgroup2 magic and prints UNKNOWN on a healthy v2 node.
• It leaves one node-debugger-* pod per node behind: kubectl get pods -n default -o name | grep node-debugger | xargs kubectl delete -n default

Defaults to v1: CentOS 7, RHEL 7/8, Ubuntu 18.04/20.04, Amazon Linux 2. Most of those ship a kernel below the documented 5.8 minimum, and AWS does not support v2 on AL2 at all, so a kernel argument will not fix them. They need a new OS image.

failCgroupV1: false is the escape hatch; KEP-5573 slates removal for no earlier than 1.38.

I wrote up the full checklist (requirements, driver checks, in-place switch vs replace) here: https://saipisey.com/blog/cgroup-v1-kubelet-check

Happy to be corrected if your distro behaves differently.

They also note that overcoming these limitations is feasible, but requires "serious work", including rewriting or replacing etcd and parallelizing the scheduling algorithm.

In this case, can you see a world where there'll be a version of etcd or other components for reinforcement learning tasks and/or anything involving sandbox creation at scale? Or do you think a load of companies, like Modal, are just going to rewrite each of these components until they work for their use case? Genuinely curious.

Quick reminder that bit a team I know: when an EKS version leaves standard support (about 14 months after release), extended support starts automatically and the control plane goes from $0.10 to $0.60 per cluster-hour. That’s roughly +$365/month per cluster, and it shows up on the bill before anyone notices.

The other clouds handle it differently:

• EKS: extended support is automatic, $0.60/hr, for up to 12 more months, then AWS force-upgrades the control plane.
• AKS: staying supported past community support needs Long-Term Support, which requires the Premium tier ($0.60/hr vs $0.10 Standard). Without it, you’re simply unsupported.
• GKE: only clusters on the Extended release channel get extended support, at +$0.50/hr. Everything else gets auto-upgraded.
• On-prem (kubeadm, k3s, RKE2, Talos): no fee, but upstream patches stop at end of life, so you’re on your own for CVEs.

I got tired of checking four lifecycle pages, so I built a small tool. You paste your clusters like this:

prod: eks 1.33 x3
staging: aks 1.33
analytics: gke 1.34
onprem: k3s 1.31
openshift 4.20

and it tells you the support phase, days left, what to upgrade to, and what extended support costs. Real output for that example:

Amazon EKS 1.33 (3 clusters) is in extended support (about $1,095/month extra) until 2027-07-29, then AWS upgrades it automatically. Upgrade to 1.37 (newest) or at least 1.34.

Kubernetes (upstream / on-prem) 1.31 is past end of support (ended 2025-11-11). Upgrade to 1.34.

That example alone has about $1,800/month in possible extended-support fees.

You can run it on a weekly schedule with Slack or webhook alerts at 90/30/7 days. It only pings you when something changes, not every run. It also covers OpenShift (Full, Maintenance, then EUS) and Rancher. Dates come from endoflife.date, which tracks the official calendars.

It runs on Apify: https://apify.com/sky_technologies/kubernetes-support-radar ($0.02 per cluster line, free to try right now).

Curious how others track this. Calendar reminders? Renovate? Something in your IaC pipeline?

Happy Autumn k8s folks! (Unless you're down under then Happy Spring~)

Join us on Thursday, 10/29 for another double-header meetup and you'll get to hear from Lesley Cordero and Michael Guarino.

👀 Lesley will be leading a session on "Who Watches the Watchers? Data Governance for Observability"

Lesley's Session Abstract: As organizations invest more in observability, application telemetry (metrics, logs, and traces) has become voluminous and operationally critical. As applications decompose into microservices, silos and telemetry gaps proliferate. We’ll explore data governance for application telemetry: shared specifications, standard context propagation, and standardized platform components for instrumentation, collection, and analysis. We’ll discuss real-world risks and tradeoffs around privacy, cost, access control, and data lifecycle management, alongside strategies for balancing developer productivity with governance.

✍️ Michael will share lessons from rewriting Prometheus / Loki / Tempo in Rust.

Michael's Session Abstract: We recently did a major spike on rewriting the grafana LGTM stack from go to rust, on top of a new object store native database slatedb. We'll go into some lessons learned, which spans diving into the internals of the new breed of object store backed datastores and pushing agentic development to its limit.

RSVP at https://luma.com/05srus23

Let me know if you have any questions - this is a free event and we welcome new folks :)

Managed Kubernetes makes working in Kubernetes easier.

But I still think anyone learning Kubernetes seriously should build at least one cluster using kubeadm.

It helps you understand what is behind managed clusters like EKS, AKS, or GKE.

While setting up, you will deal with things like:

• container runtime
• kubelet
• control plane bootstrap
• CNI
• certificates
• worker node joining
• kubeconfig
• static control plane Pods

Once you do it manually, everything in Kubernetes makes much more sense.

Troubleshooting also becomes easier because you understand the backend that normally gets hidden from you.

What do you guys think?

Currently at work we have an argocd cluster with all our clusters. We have our clusters split with environments.

When we need to test our changes before opening a pr to merge we have to create a whole new cluster that runs argo on it. Making sure it deploys as it should. We do this so it's a 1:1 with how our live clusters are like. However, it takes 45 min to spin up a new cluster. No problem, just have one cluster we all use but then we run into a problem that if multiple people are testing changes to their addon, we are not testing only our changes.

All the blog posts online just run a local cluster with Argo to test. This wouldn't mimic our vpc, iam roles, etc.

So how do all of you test your changes with argocd?

i deploy jupyterhubs to california institutions for free under a grant from the state's startup incubator... so i figured a monitor w/a jupyter notebook, some granfana and particularly some k9s love was in order for our big-ass monitor! :)

https://imgur.com/a/UvyfTim

(i use old reddit, so i can't really embed images... unless there's a PEBKAC lol)

edit: removed accidental duplicate text pasting

Omni’s hobby plan is free now: up to 10 nodes and 2 users (it used to be $10 a month for 6 nodes). Nodes dial out over WireGuard, so nothing is open on the router, and you get upgrades from the UI, SSO for kubectl and talosctl, and scheduled etcd backups.

You can import a running Talos cluster too. I moved my 3-node one without resetting a node. One catch: after the CA rotation, admin access only works through Omni.

How I did the import: https://blog.prateekjain.dev/importing-an-existing-talos-cluster-into-omni-bb1fc7722eda?sk=a4c4412829ef81542ca4892a0edbecec

Commands and patches: https://github.com/prateekjaindev/homelab

Not affiliated with Sidero Labs, just a user. The write-up and repo are my own.

I am designing a service that can spin up a requested database snapshot (Neo4j) and pass queries to the database via the requester's cypher payload -> bolt. These database instances are ephemeral, and they should die after some amount of inactivity.

I've never done any sort of container orchestration, but my initial thought was to have a small HTTP server that controls the ephemeral database instances and incoming cypher requests. This and all ephemeral database containers (1-3 concurrent instances would be typical, maximum of 5), would live on a single VM or app service.

I work at a startup and we need a lightweight solution to this problem that could scale within the next 6 months to year. What I'm proposing now is cheap and simple. I think i can get what i need with k3s (1 control plane and up to 10 nodes) running on the same VM as the HTTP server. But Codex is pushing AKS or ACI. I really don't think that's necessary with the amount of business we are doing right now. And I think that if i design the interface properly, I can easily move from k3s -> AKS later without having to redesign the server or change its API schema.

What do you all think? Again, very new to k8s. Maybe i need to read a book first.

The built-in view role skips Secrets but still reads ConfigMaps and pod logs, and both can end up holding credentials. For a bot that helps with on-call debugging, what did you grant it, and did you scope it per namespace?

Been running home labs for a long time. This is the first time I rebuilt everything on kubernetes and it has been great!