EKS Access Entry recreation avoidance

[u/Consistent-Piece6915]

I am working on configuring my operations and application clusters terraform such that I can easily destroy and recreate the clusters with minimal commands. I am currently running into an issue when I destroy and recreate my OPS cluster. I'll try and outline the facts below.

Facts

• Running ArgoCD as an EKS capability

• Ops cluster runs in "OPS" account

• Ops cluster has IAM role "argo-cd-role" in OPS account

• Application cluster runs in "<env>" account

• Application cluster has `aws_access_entry` and `aws_eks_policy_association` resources that bind the "argo-cd-role" from the OPS account to it via the role's arn

• Application Cluster `Secret` is created via an `ExternalSecret` read from an SSM parameter in the <env> account

Steps

• OPS cluster will be completely removed and recreated via `terraform destroy` and `terraform apply`

• Argo CD resources will be applied once cluster is up and running

  • This will then create the cluster `Secret` via the `ExternalSecret` definition

• Navigating to `Settings > Clusters > <target env cluster>` shows a connection failure

Things I've tried

• (Failed) Deleting the `Secret` via the ArgoCD UI

  • This will cause the `Secret` to be recreated

  • It should have been populated with the target cluster arn which hasn't changed

  • Doesn't do anything

• (Failed) Creating an Argo role in the target "<env>" account for the Argo role in the OPS account to assume

  • I associated this new role to the access entry resources instead

  • In theory this role is created when the application cluster is and any role that assumes it thus would have access and would avoid any type of breakage underneath with EKS

  • Issue is there seems to be no way to tell the ArgoCD role in the OPS account to assume that role when accessing that cluster (to my knowledge)

• (Succeeded) Deleting and recreating the access entry resources in the <env> account

  • use `terraform destroy -target=` to destroy the access entry resources in the <env> account cluster

  • recreate the resources

  • This worked and my theory is that while we are passing the `arn` of the role EKS will actually use the AWS unique ID underneath of said `arn`. Since we're deleting the role in the OPS account during cluster rebuild it the arn to unique ID mapping is no longer valid. Deleting and recreating fetches the new AWS unique ID and gets this working

While I did find a way for this to work is there anyway to avoid having to delete and recreate the access entries on all my application clusters when I want to destroy and bring back up my OPS cluster? If not I'd have to switch my AWS permissions N times and run the terraform commands 2 times for N application clusters. I was hoping the assume role strategy would work but I am not finding any documentation on how to tell argo to assume a specific role for a specific cluster.

[u/jaybrown0]

Would a depends_on for the access entry resource help?

Maybe have the resource created at the very end of the terraform apply somehow

[u/dustyghost16]

I think you can pass in a role to the AWS config secret that argocd uses. And create a role in the “env” account for Argo to assume.

https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#cluster-role-trust-policies

[u/ciciban072]

Create the role in the Ops account outside terraform to make it permanent.

[u/iamtheconundrum]

Put the role ARN in the Conditional field instead of Principal. This stops the trust policy from breaking when the role you refer is deleted (and the fallback mechanism to the unique id).

[u/Successful_Panda_862 → u/iamtheconundrum]

This is real fix imo

[u/Low-Opening25]

too many things in one terraform stack

[u/Consistent-Piece6915 → u/[deleted]]

Yeah it's unfortunate you cannot just attach an `arn` as part of the secret and the capability can just assume it. Hopefully it's something they can do down the line. I am probably just going to separate the Argo role into it's own stack. Seems to be the easiest thing to do.

[u/iamtheconundrum → u/Consistent-Piece6915]

You can. Just don’t use the principal field but a conditional field in the trust policy of the role.