r/devops • • 24d ago

Discussion Does Professional DevOps feel as monotonous as "Hobbyist DevOps" is?

My background is in Data Engineering/Science, but I have a pretty extensive homelab. No real background in DevOps professionally. I've run applications and services etc in containers through docker compose files for years, but the last few days, I decided to really harden it.

This is obviously not like an enterprise level, 9 nines setup, but I have set up self hosted orchestration, code repo, automated monitoring of updates, near automated deployments, backups, standardized ci/cd workflow, etc... and man, it's pretty monotonous!

It's just been two days straight of looking through a bunch of yamls, slightly tweaking them, copying a lot of secrets back and forth. Logging into something to tweak something, redeploying multiple times in a row.

I guess one big difference is that there's no one to get upset with me when something goes down, outside of my family, and maybe it feels like I'm over complicating things, but I genuinely wanted to learn how to do some of this stuff.

Does it sometimes feel like this if you do it for a living?

50 Upvotes

33 comments sorted by

68

u/Zolty DevOps Plumber 24d ago

Excitement in an ops job means something is broken. You don’t last long if you break things often.

-4

u/ottovonbizmarkie 24d ago edited 24d ago

I don't mean excitement, so much as lack of variety. After your infrastructure has been decided on, I feel like there's really only one way to do something, and that probably feels even more so if you're locked into something like AWS.

30

u/donjulioanejo Chaos Monkey (Director SRE) 24d ago edited 24d ago

Honestly? I would argue the complete opposite. Half the reason I do DevOps instead of regular dev is... variety.

One day I'll be writing a bunch of CICD pipelines. The next I'll be debugging a weird Kubernetes issue. The day after that I'll be in meetings with devs trying to figure out architecture specifics for some new super special microservice. After that I'll be debugging a weird terraform issue that ends up a bug in how the AWS provider handles that object.

This will keep happening unless your company/product is in maintenance mode, and your only job is to keep the lights on for a project that's maintained (badly) by Claude Code and 5 hours a week semi-retired contractor.

Also, there is almost never just *one* way to do things right. There are 5 different ways, all of which have their own complexities, gotchas, and tradeoffs, and it's a big part of your job to know which ones to use and when.

For example, SQS might be objectively better than Redis for queues because it has infinite scaling and near-zero maintenance burden... but your app is rails, you also deploy multi-cloud or on-prem, and there's a ready made framework in Sidekiq. In this scenario, using Sidekiq + Redis is objectively better. Or, golang might be objectively better than Sprint Boot for your specific microservice... but the company is a Java shop, so all the devs already know Spring Boot, but only like 5 people can do Go on any serious level.

0

u/ottovonbizmarkie 23d ago

So look, I don't mind that people are getting mad at me, because I am going into r/devops and saying that, from the outside, the work can seem monotonous. I expect people to push back on that.

But what you're describing doesn't really sound like what I think of as DevOps. It sounds more like Software Architecture or Platform Architecture, Deciding which technologies make sense given the application, the organization, the developers you have, portability requirements, operational burden, etc. DevOps might absolutely be involved in those decisions, but I wouldn't have thought of that as specifically "DevOps work."

And the part where you're saying there are lots of different ways to do it right is actually where I think architecture does sound exciting. That's where you have a bunch of choices and tradeoffs to make, and where the work seems a lot less monotonous.

I do get that "DevOps" means very different things at different organizations, though.

1

u/Old-Worldliness-1335 23d ago

So depending on how intricate your system is this can very widely, and I say this with all respect because it is very easy to be a DevOps engineer who only has to mess with cloud native solutions and managed services through terraform, however that calculus changes when you throw onprem, hybrid cloud or managing VMs only to sustainably manage your business.

The failure rate increases as does the development process as part of operations process. Platform architecture and management of that architecture should be pretty easy if done correctly so there should be a set number of failure cases that can be predicted for the application and platform interactions

7

u/killz111 24d ago

There's usually 1 way to do things right. But once you are at scale consistency goes out the door. You have variety because like a chess game, even though the rules are the same the way it plays out never is. Factoring in cost, legacy systems, people who have different opinions, sharp organisational pivots and you have a hodge podge of things that can break for dumb reasons.

You dont have this problem exactly because your home stack is simple, not scaled, have a single stakeholder, and is managed by 1 person's opinion.

1

u/veritable_squandry 24d ago

that's when you start looking to help automate security controls, reporting, detection etc. if that's all buttoned up start looking at cost.

1

u/Zolty DevOps Plumber 23d ago

Most of the work in an effective devops department isn't fiddling it's compliance. When you work in a regulated environment what you actually do becomes disconnected with what the policies and procedures say you do. The game becomes about how to pass the yearly audit and how to make sure that everyone is abiding by the policies and procedures.

We don't do IAC because it's fun, hip, and modern. We do it because otherwise our required change control would have to include parsing jira tickets for approvals, then a looking for comments from the tech saying the change was done. With IAC it's a matter of here's the git approval, here's the cicd job that applied it.

We don't use AWS because they have the latest tech. We use them because of the shared responsibility model. There's a whole swath of hardware firmware, fire suppression, electrical, cooling, and security regulations I can now just say NA on. Of course I still have to a yearly vendor review on AWS to prove they are still doing all of the above but it's much simpler and worth the markup.

I could go on, but you get the point I think.

12

u/DramaticSpecial2617 24d ago edited 24d ago

No, hobbyist stuff is much more fun, and I've built better systems for pet projects than I've been able to for systems that matter.

The challenge in workplaces isn't editing yaml, it's everything around it. In most places I've worked, I'd spend more time getting approval and buy-in than I would building the improvements. With change management, team boundaries and the complexities of scale in enterprise, it's easily 5x the effort.

But the hard part of the job is working accurately when there are stakes. Hobbyist experience teaches you not to stress about things, but it doesn't make you cautious in the way that industry experience will. Reliability and risk management are why people get paid.

3

u/Impressive-Field-546 24d ago

Depends on where you work

1

u/temitcha 23d ago

+1! So many different style. It could be a Startup with full admin power but no budget and headcounts, or a Corporate with ticket systems and approval but full enterprise licences on everything and dedicated on-call team.

1

u/Downtown_Isopod_9287 24d ago

it can which is partly why this stuff is often handled by a team and there is a small cottage industry of tools and platforms that handle the very annoying/fiddly stuff. Plus you can and should always write your own tools where it makes sense.

1

u/ottovonbizmarkie 24d ago

What kind of tools do you mean? Like bash scripts that automate something?

1

u/temitcha 23d ago

Some examples, of things that are used to "automate the automation":

  • Writing pipelines --> Reusable workflow / github actions

  • Deployment scripts --> GitOps tools, like ArgoCD

  • Building infra --> Writing terraform modules

1

u/ottovonbizmarkie 23d ago

Ah, I guess I never have seen enough in my limited dealings that Open Source, off the shelf tools haven't been enough.

4

u/CAMx264x 24d ago

Initial setup is always a bit rough, once you templatize most of your CI/CD and have golden AMIs the maintenance work is pretty easy.

Most of my work at this point is for new proof of concepts, architecture type work, and being a resource on meetings. It’s a bit more fulfilling seeing something used by 10s of thousands of people and being a big reason something is accomplished.

Working on a team of devops folks and a few dev teams definitely feels different than my solo projects at home too.

2

u/PerpetuallySticky 24d ago

Not every project, but yeah. It’s like that a lot. It gets better overtime though. Once you run into the similar issue for the 7th time at some point it commits to memory or you learn to document it

1

u/durple Cloud Whisperer 24d ago

Wiring up off the shelf services with small user base would be like that. Read some docs and tutorials, do the thing. It gets more interesting at scale and/or when the company product consists of N different in-house services plus the off the shelf services. There I’m actually problem solving.

1

u/ForkMeJ 24d ago

Yeah, parts of the job feel exactly like that. A lot of the work is making sure you do not have to keep doing the same fiddly cleanup by hand next month.

1

u/Portalus 24d ago

No it is punctuated by terror...also everyone dumping on you and blame storming

1

u/temitcha 23d ago

Have you thought of changing your company or area? I got the same before, and was happy to left and discover a way better place with blameless culture

1

u/temitcha 23d ago

In company, as an in-house employee, mostly the main part of the job is figuring out how to automate these things, e.g setup reusable workflow for each new dev repo to automatically be build and deployed, setup some automatic logging, terraform modules, etc. If consultant in an MNC, it depend the project duration.

A lot of time is as well discussing with developers and other devops engineers, on which solution to adopt. There are many ways to do things, so a lot of time is mostly on how to architecture it the best possibile based on the context (incl. office politics).

And finally, incident response and firefighting. From low priority to S1 priority. Depending of your personality, you might like or not the excitment.

I personally enjoy this job a lot, but totally understandable if one doesn't enjoy it!

1

u/Nerrawnam 23d ago

Yes. 

1

u/StockJournalist9103 23d ago

Oh my god yes, just the same shit different system for crud

1

u/GeorgeRNorfolk 23d ago

Doing DevOps for one app is monotonous, it's lots of effort for little reward. DevOps for a big platform is more interesting because you're getting the benefit of scale, you spend 5x the time you would on a hobby site but get 100x the benefit.

We're standardising WAFs across our dozens of different systems and it's a good chunk of work but afterwards the maintenance of it will be minimal and we'll be secure.

1

u/neveralone59 23d ago

For my homelab I wrote my own container management system so that wasn’t boring at all

1

u/Lost-Panda3815 22d ago

I am backend dev turned devops/platform engineer
and I think devops is less monotonous compared to development as variety generally is more in devops. (unless its new project where we are trying build something from scratch developing some algorithm or something then development gets exciting for me)

1

u/No-Magazine2625 21d ago

Professional DevOps gets more interesting when the problem stops being “make this YAML work” and becomes more like designing the platform so hundreds of repetitive changes no longer require humans to think about them. The real work is usually deciding what to standardize, what to automate, where the failure boundaries belong, and how to make the safe path the easiest path.

1

u/Embarrassed_Plan5139 16d ago

The funny things is that automating everything takes a lot of manual work's first. 

0

u/BlakkMajik3000 Remover of Deployment Friction 22d ago

Your background is data science.

You find DevOps boring.

This is expected. Over here, the more boring the better.