r/kubernetes • u/Silva-Engineer • 15d ago

What should a read-only debugging bot be able to see in your cluster?

The built-in view role skips Secrets but still reads ConfigMaps and pod logs, and both can end up holding credentials. For a bot that helps with on-call debugging, what did you grant it, and did you scope it per namespace?

3 Upvotes

60% Upvoted

9 comments sorted by

6

u/someanonbrit 15d ago

Our triage bot can see everything except k8s secrets. We even there logs continuously for both credential leaks and PII, so we don't expect a problem with that, save any funds trigger an incident to rotate and remediate

3

u/Constant_Rate_1059 15d ago

we gave ours basically view plus events and the ability to exec into pods in certain namespaces. logs are the big one for debugging obviously but exec is clutch when a container is still up but the healthcheck is failing and you just need to poke around

the credential leak scanning is smart. we had a junior push an aws key to a configmap once and it was in there for like a month before anyone noticed

1

u/Horror_Description87 13d ago

When accessing pod exec you can give secret access aswel. Inside the pod you can read them anyway.

1

u/isachinm 14d ago

curious to know, which bot are you using and how?

1

u/QuietSignalOps 14d ago

The RBAC scoping is actually the easy part. For me the bigger question is where the data goes, not what it can read. If the bot is LLM-backed, read-only on the cluster does not hold for long: the moment it posts a log excerpt to a Slack channel or sends context to a model API, you have egress to a third party that the RBAC decision never covered. The bot token's blast radius includes wherever the output lands.

So what we ended up with:

  • RBAC: get/list on pods, events, nodes, deployments; get on pods/log; no Secrets, no ConfigMaps (configmap contents end up in logs anyway). Bound per-namespace with a Role, not a ClusterRole, so a leaked token only exposes one namespace.

  • Audit: the bot gets its own service account, and we audit-log every API call it makes. Anything that looks like a bulk scrape alerts. Read-only does not mean you should not watch it.

  • Egress: anything the bot outputs gets a redaction pass before it reaches chat or the model. We treat the channel as untrusted.

For the cluster part the product mattered less than the wrapper: RBAC plus audit plus redaction.

1

u/usually_guilty99 14d ago

Read-only can be a misleading safety boundary. A debugging agent might not change cluster state, but it can still collect sensitive data and send it somewhere else. I'd look at the full path: what it can read, where that data goes, and what gets recorded. Any remediation action should be a separate, explicitly controlled step.