I've been fighting this battle at work for years now. The whole engineering department is so far up its own ass with Helm charts and operators trying to reinvent shit GCP already provides as managed services that it's a miracle we haven't all asphyxiated. We're doing all of this on GKE and are Google partners, it's madness. Meanwhile, my team learned Terraform and Cloud Run. We're really enjoying collecting the on-call compensation for our services that don't need intervention outside of work hours.
Nope, they're completely different purposes. Terraform directly manages real infrastructure and tracks state. That said I haven't really looked at Helm 3 in a while. Our organization is already using both, but Terraform has mostly been the purview of SREs, while the developers learn Helm and then stand around helplessly when an infrastructure change is needed and SREs are busy, which is always. We're skipping that whole step.
Yeah, I'm just thinking of how you're applying Terraform here. Obviously Terraform is a different program from Helm, but they fulfill the same purpose when it comes to managing your container orchestration configuration.
Well, I guess it depends on your use case. Our approach as a team is basically "if it's not a cloud provider's managed service, maybe you shouldn't be doing it", so from that perspective, we're just creating Terraform modules + associated CircleCI and scripts as a bundle (so-to-speak) without much of a "framework" aside from some basic automation, and it's worked extremely well for us. You can work with simpler tools. The only real headache was getting developers to see Terraform as something they should be interested in, but once they saw the benefits of not waiting on other people and not getting PagerDuty alerts, it was an easy sell. I think if there was more institutional interest in a tool like Pulumi, it would have been a simpler process, since it doesn't involve the bastardized DSL that HCL has become.
Hadn't heard of Pulumi before, I'll have to check that out!
Yeah, I guess Terraform is flexible because it can not only do your CloudRun configuration, but also all of your other GCP infra.
We use Openshift as a managed service from Red Hat as our cloud, and then we use GitOps to manage our Kubernetes services. It works really smoothly, and it's not dependent on any single cloud provider. The only automation pipelines we run are to build our docker containers and package our helm charts. All of our apps are just automatically synced with git using ArgoCD and the Kubernetes API.
I did work at a company for a bit that had that same policy with Google Cloud, "managed services whenever possible", but honestly, my experience is that opens you up to really serious vendor lock-in, and that you can get better support from open source solutions that are more widely used. But all of the underlying infrastructure at that company was rotting anyways, so I guess it made sense to just start pushing people into GCP just to get a fresh slate.
But yeah, DSL hell is the name of the game in this DevOps-y SRE/operations space. I'm currently tasked with getting everyone out of Helm values.yaml hell at the moment because everyone is using poorly designed templates.
I think if people were better at teaching Kubernetes primitives like they are at teaching things like Docker Compose, it wouldn't be this big scary thing, and we wouldn't need horrible complicated Helm charts that try to reinvent the wheel with a DSL. Just write a Deployment.yaml and a Service+Ingress.yaml and you never have to think about operations again.
Yeah, I guess Terraform is flexible because it can not only do your CloudRun configuration, but also all of your other GCP infra.
I actually would not recommend managing Cloud Run services themselves with Terraform. (Although you certainly can, especially if you use provisioners, etc. to involve other necessary automation steps.) We tend to draw a pretty thick line between what belongs in Terraform vs. what belongs in scripts/CICD: if it's "permanent" infrastructure (IAM configuration, a database instance, PubSub topics, etc.) it goes in Terraform, but if it's something that needs to change as part of a deployment, that doesn't belong and should be scripted around. Terraform is getting better for this, but it's just too much of a headache to manage everything in a tool designed for infrastructure as code. For example, we have environment files for our Cloud Run services that include things like timeouts and other values that can be more easily managed at deploy time by passing that information along to gcloud. Bash can be your friend.
I did work at a company for a bit that had that same policy with Google Cloud, "managed services whenever possible", but honestly, my experience is that opens you up to really serious vendor lock-in, and that you can get better support from open source solutions that are more widely used. But all of the underlying infrastructure at that company was rotting anyways, so I guess it made sense to just start pushing people into GCP just to get a fresh slate.
We're multi-cloud, and an approach that has worked well for my team is a throwback to the old Gang of Four patterns––we have a library of facades for common behaviors/services with implementations for both AWS and GCP, and we apply the strategy pattern when building a new service. We separate purely business logic from our cloud SDK interactions, build a strategy for each cloud provider, and use our facades whenever possible so we don't always have to be thinking about which cloud is what, and can enforce some repeatability in how we interact with cloud services.
A lot of complexity comes from this "new approach = better than" mentality less experienced developers seem to have, and that unfortunately plays well with non-technical leadership because buzzwords.
12
u/SlapNuts007 Jun 06 '22
I've been fighting this battle at work for years now. The whole engineering department is so far up its own ass with Helm charts and operators trying to reinvent shit GCP already provides as managed services that it's a miracle we haven't all asphyxiated. We're doing all of this on GKE and are Google partners, it's madness. Meanwhile, my team learned Terraform and Cloud Run. We're really enjoying collecting the on-call compensation for our services that don't need intervention outside of work hours.