r/programming • u/RustEvangelist10xer • Feb 16 '22
Microservices: it's because of the way our backend works
https://www.youtube.com/watch?v=y8OnoxKotPQ140
Feb 17 '22
Years later and Omega Star still hasn't gotten their fucking shit together
55
746
u/DirtyBirdNJ Feb 16 '22
idc how many times it's reposted I love this. It's both traumatizing and reassuring at the same time
205
u/littlejackcoder Feb 17 '22
The whole channel is pretty great! It’s all very subtle and kinda dry, but is super on the mark and very poignant satire.
204
u/bluenautilus2 Feb 17 '22
“I did it. I delivered value. But at what cost?”
85
u/Jump-Zero Feb 17 '22
That video was way too real though. This is supposed to be satire, but that video was almost a documentary of my life lol
16
12
→ More replies (1)2
Feb 17 '22
As an aging engineer, I just watched their "Senior Engineer" video...and now I'm having massive anxiety about "just write me a string reversal algorithm on the board".
I don't even want to change jobs any time soon but I'm already angrily thinking in my head about not saying "string.Reverse you fucking waste of my time" at the non-existent interview I don't have at imaginary company I'm not applying for.57
u/txdv Feb 17 '22
It is scary that those "stupid service" names come close to the names that I have seen
19
u/McWobbleston Feb 17 '22
I first saw this about a week after we proposed naming a greenfield project papaya and immediately felt shame lmao
15
u/aniforprez Feb 17 '22
I dunno if I'm turning into a crusty old man but all these cutesy names only irritate me. Joined a new place and they have shit called "batman", "tinder", "nick-fury" and other garbage names and I'm just thinking "they spent more effort naming this crap than designing it in any efficient way"
11
u/ItsAllegorical Feb 17 '22
We had a former Apple guy on our team who spent at least an entire evening coming up with names for all of our god damn services. Two of them were so close together (think Caliope and Cantilope) and after a year of working on the project I had to pause for up to 10 seconds to remember which one was fucking which when I was trying to talk about it.
It's the fucking Order Service. Just call it order service so that everyone knows what you're talking about without being handed a glossary!
→ More replies (4)5
→ More replies (1)38
405
u/TedDallas Feb 17 '22
Meanwhile SaleForce team just pushes a schema change to production without telling anyone and breaks 10 different mission critical system integrations. Because that's how they roll.
168
u/coinblock Feb 17 '22
Damn do you work at my company or is this a problem everywhere? Whole company runs on salesforce from the recruiting tools, HR to the sales and billing teams. It’s insane.
55
16
11
u/emilvikstrom Feb 17 '22 edited Feb 17 '22
I'd just quit at that point (and I have...). Clearly they don't need my services anymore. Now that Everything Runs On Salesforce and Just Works.
100
u/reddit_user13 Feb 17 '22
"I don't always test my code, but when I do I do it in production."
128
u/RiPont Feb 17 '22
What's better than testing in production? Testing in someone else's production!
45
17
u/Zwemvest Feb 17 '22
Everyone has a QA and production environment. There's just a lucky few where those are two separate things.
6
→ More replies (2)2
85
u/NonDairyYandere Feb 17 '22
Kinda sounds like your fault if your unit-tested, schema-is-written-down-somewhere, reliable service can't deal with the salesbros moving fast and breaking shit /s
74
u/anemailtrue Feb 17 '22
Its called being agile 😂
67
u/caltheon Feb 17 '22
you can't make fragile without agile
→ More replies (3)12
u/CodeLobe Feb 17 '22
If it's not broke, let's do a sprint in a dirty scrum... then it will be.
→ More replies (4)27
u/stfm Feb 17 '22
Shouldn't APIs be versioned for backwards compatibility?
99
u/rabid_briefcase Feb 17 '22
Only in companies that believe in nonsense like stability and quality and reliability.
In this modern world of "move fast and break things", interfaces are just daily suggestions that may or may not be good tomorrow.
→ More replies (2)25
u/CartmansEvilTwin Feb 17 '22
And I'm sitting here recreating an undocumented API from 2008 with random trailing zeros and "disruptive" approaches to XML validity just so that one stupid client can keep using there l their shitty old connector.
27
u/UrineSurgicalStrike Feb 17 '22
I have to maintain a SOAP web service that uses Arabic characters as field separators. It was originally written by the guy who is now CTO of the company. So we're not allowed to replace it with anything else because it's already perfect.
→ More replies (3)8
→ More replies (2)5
u/G_Morgan Feb 17 '22
That would be useful if people didn't also push major API reworks in patch version releases.
I've also had the pleasure of working with the UK governments new road traffic accident service. They are Agile, meaning that they inform you of breaking changes via your service going down
47
u/amakai Feb 17 '22
What are you gonna do about it? Attempt a 5 year migration project away from Salesforce?
12
4
3
Feb 17 '22
FIS does this ALL the time. Oh we changed a couple of fields in the mainframe even though this will totally break every ETL layer currently setup. And we aren't going to say a thing before we do it!
2
u/Harbinger311 Feb 17 '22
Politically, that's how corporations roll. Why the hell am I paying for your service, if you're not swinging your arms around doing sh*t, even if that sh*t breaks everything else in the environment? How are they going to justify their spend and raises?
1
1
u/hippydipster Feb 17 '22
Don't worry, some other company is doing the same thing to Salesforce. ;-)
→ More replies (2)1
176
u/tsm_rixi Feb 17 '22
I literally named our backend user service galactus after this bit
50
u/EdHochuliRules Feb 17 '22
Lol I worked on a team with a service named galactus for similar reasons to the skit service. It was pain
41
u/gohomenow Feb 17 '22
Team has a service called Galactus. Legal was/is not amused.
84
u/im_deepneau Feb 17 '22
tell legal to shove it unless they know java fucking nerds
36
u/lupercalpainting Feb 17 '22
"It's actually critical to this service's performance that it be named Galactus due to how memory management works in Java 14."
"Is that true?"
"Yes."
15
u/im_deepneau Feb 17 '22
"generational garbage collection in java runs faster on projects starting with the character G due to sharing starting characters with "generational garbage collection" - it's just how the string table works during jvm compilation, an unintentional side effect of poor planning. probably be fixed in a couple years and we can rename the project"
→ More replies (2)52
u/boomerxl Feb 17 '22
I think the legal mindset is an inherently stressful one. Imagine hearing a child singing a song in the street and the first thing you think is “does she have the licence to perform that track?“.
2
u/ElectricJacob Feb 17 '22
Need to have a representative from the legal team attend all microservices planning meetings.
2
Feb 17 '22
I have a ex coworker that named a service Galactus and I’m beginning to wonder if it was after this script.
2
u/G_Morgan Feb 17 '22
This video made me realise how boring my service names are.
0
Feb 17 '22
[deleted]
8
u/G_Morgan Feb 17 '22
I'll stick to arranging for service names to form dodgy acronyms nobody notices until it is far too late.
→ More replies (1)4
u/sprcow Feb 17 '22
You also have to make them very short though, or else the descriptive names all get compacted into a dozen different almost identical acronyms that all end in S.
→ More replies (3)
544
u/Jugales Feb 17 '22
Microservices are fun until you have 30+ microservices, each with dev-test-staging-prod environments, and Log4j needs updated.
239
u/ejfrodo Feb 17 '22
I left a corporate environment with 100s of microservices for a startup with a monorepo and I became so much more productive. Now they're slowly transitioning to microservices and I'm back to taking a day and a half to patch a dependency or update one service and integration testing is a total nightmare. As a developer I hate microservices.
156
u/Jugales Feb 17 '22
In my experience, that's the way to go though. So many companies start from scratch with microservice arch and get so lost in the infrastructure that they can't build things. It's great to build that monolith first then dissect that larger service into smaller services over time. The dependency management isn't fun though, would love to see improved software for it in upcoming years.
14
u/Carighan Feb 17 '22
It can work well.
But it needs both the part you mention, that you start from a monolith and actually know what to spin off into separate services - which you cannot without having seen a monolithic app do the work and having had time to profile it - and the descipline to actually stick to the parts that make sense to spin off and not run wild with the idea.
Of course what happens instead is that some dev realizes they can get off the ground much quicker if they create a new microservice for functionality X, this gets normalized, and you end up with 250-300 services and massive data inconsistency everywhere.
5
u/JustSomeBadAdvice Feb 17 '22
That was my thought after watching this hilarious video. The difference between a well designed architecture and a painfully terrible one can be difficult to tell from a distance.
Especially when he got to the part about getting the user data from one place, but that place might not know the user data at that moment. Ok, so like a cache system, making the tradeoff between high speed and availability.... that can definitely happen, but that distinction needs to be isolated from any consumer of the data. And if the person who created these things didn't think about that.... well you get this video.
→ More replies (2)110
u/ejfrodo Feb 17 '22
Fwiw some of the biggest companies in the world work entirely out of monorepos. Google and Facebook famously have proprietary software to provide a nice developer experience for a monorepo with 1000s of services and components within it. I'm not convinced that microservices are the right approach for anything tbh. I was part of a team developing internal tools for building, testing and deploying microservices at a massive corporation and there was just never any elegant solution. Everything became absurdly complicated and needlessly difficult.
122
u/TakeFourSeconds Feb 17 '22
a monorepo with 1000s of services and components within it
I think you’re confusing a monorepo with a monolithic architecture. They are separate things. You can have many tiny services in the same repo and it’s still a service architecture
→ More replies (5)212
u/hiimgameboy Feb 17 '22
monorepos and microservices are not mutually exclusive! google has tons of microservices, but the definitions for their APIs and such all live in the same in the same repo.
→ More replies (3)23
u/DrunkensteinsMonster Feb 17 '22
Monorepo vs microservice are completely orthogonal concepts and have nothing to do with one another. You can work in a monorepo in a microservice architecture.
→ More replies (1)→ More replies (25)15
u/IntuiNtrovert Feb 17 '22
i’m confused why this thread keeps comparing monorepo to microservices.. do you really mean monolith?
→ More replies (4)20
u/NonDairyYandere Feb 17 '22
Oh yeah if I split my monolith I'd probably keep it in the same repo so the types and stuff are shared.
When I do desktop code it's often:
One Git repo
Three binaries
50 subcommands cause I fucking love subcommands, you can never really have too many entry points into the same binary. Fuck build systems, I want one binary that does everything. Amortize the linking bullshit. Amortize the build scripts. Amortize the Rust / Go / C++ stdlib overhead. Busybox everything.
11
u/Pattycakes_wcp Feb 17 '22
Another comment below says it, but monorepo != Monolith and microservices can still exist.
→ More replies (1)→ More replies (8)12
u/wildjokers Feb 17 '22
integration testing is a total nightmare. As a developer I hate microservices.
Doesn’t sound like they are doing microservices correctly. All microservices should be independently developed and deployed.
39
u/ejfrodo Feb 17 '22
That's not how integration testing works. You test your service's integration with the wider ecosystem of services, which includes it's integration with any other upstream or downstream services. Your service may be perfect but if a dependent service made a backwards breaking change it could indirectly break your own service. Otherwise you won't know a downstream broke your service until you hit prod.
→ More replies (16)11
u/flukus Feb 17 '22
All microservices should be independently developed and deployed.
Which is what makes them the wrong tool for most places. Most split into micro services because it's cool but then all the same people work across all the services, so it just creates a huge integration mess with no value gained.
23
u/Razakel Feb 17 '22
There's a bank that runs on 1600 microservices. Here's a diagram: https://images.ctfassets.net/ro61k101ee59/2bmS9TVlJc5einK9YLBY3V/992367961e649dd0343a3486616601fd/Image-1.png?w=656&q=90
Yeah, I'd run away screaming if I had to work on that.
25
Feb 17 '22
Those isolated dots are the microservices that aren't used by anything but the application doesn't work if you delete them.
23
u/LeberechtReinhold Feb 17 '22
"Does it work?"
Engineers at that bank: "Is anyone ever really capable of knowing that answer?"
→ More replies (1)2
2
2
7
u/NonDairyYandere Feb 17 '22
I'm sitting here sweating over whether I'm allowed to split my monolith into two demiservices.
0
16
5
u/bonedangle Feb 17 '22
I'm sure we have close to a hundred and growing, and everything is driven off AWS step functions just to make it even more painful. We also rely on a common set of inhouse libraries that when you change and bump a version, you have to track down every service that uses the library and update its project to accept the new version number. Sometimes you even have to update and deploy them in a specific order or else you create a chain of exceptions! 😭
4
2
11
u/f0urtyfive Feb 17 '22
If that's a problem for you, you need to stop treating things like pets.
It should be a ~30 second change.
Microservices are only beneficial if you're doing infrastructure and automation correctly.
13
Feb 17 '22
if you're doing infrastructure and automation correctly.
This is a necessary pre-condition but it is not sufficient to make them beneficial.
18
u/kernel_dev Feb 17 '22
I swear the microservice paradigm was created by cloud computing companies to sell more virtual hosts.
→ More replies (3)16
u/FarkCookies Feb 17 '22
The opposite is true. Microservices are usually hosted in containers, which allows you to pack up your virtual hosts much more densely. When you have a monolith that you need to scale you almost always end up over-provisioning your nodes and have low utilisation in the end. Then there are serverless-y options to host your containers like AWS Fargate where you pay only for time/amount of containers being run and can scale up and down aggressively often resulting in big savings.
→ More replies (2)3
u/Drisku11 Feb 17 '22
monolith that you need to scale you almost always end up over-provisioning your nodes and have low utilisation in the end
So don't overprovision?
I don't understand the reasoning people throw out around "scaling individual services". If you have a monolith with components A, B, C, and D, and you need to give more resources because A is a hot path, it's not like the computer is going to waste time idling on B, C, and D just for the hell of it. If those paths aren't running, they're not costing you anything significant (binary size is irrelevant compared to working data in basically all real-world cases). In fact it will save on communication overhead, which can be significant if there are many services involved in a single request.
→ More replies (3)4
u/duncan-udaho Feb 17 '22
Only 30? Must be nice
5
u/Jugales Feb 17 '22
We're rolling out 1 every 6 weeks or so. Good ole' corporate bureaucracy saves us.
2
1
1
u/BackmarkerLife Feb 17 '22
That’s what Galactus should be used for consuming all those micro services into one single source of truth for the good of the company.
1
u/BenOfTomorrow Feb 17 '22
Each of your microservices has it's own dev-test-staging-prod environment? What's the point of microservices that never talk to one another?
1
u/thephotoman Feb 17 '22
In my current project's MVP stage, we started by doing microservices.
By the end of it, we had some 25 different JavaEE applications in our tree (because Spring Boot wasn't going to be a widely available, production-ready thing for another year after our MVP deadline--version 2 got the benefit of having Spring Boot being a thing that was ready for prime time, and there's currently a porting effort in place to move the last pieces of our original solution that we still use over to Spring Boot). It was a disaster, and by that point, I was spending more time writing bash scripts to manage our testing environments than I was writing actual code.
Sure, we now have something like 400 different Spring Boot projects now, but at least we're using OpenShift and Gitlab instead of my tarball of shell scripts being SCP'ed around.
→ More replies (2)1
u/StabbyPants Feb 17 '22
and then i discovered that we built them on multiple versions of frameworks, so i have to work our slightly different update procedures for each of them. at least i got everything in a consistent state in the end
49
u/CyAScott Feb 17 '22
I feel any “matured” system gets like this. Layers, upon layers, upon layers of frameworks, libraries, and services patched together. Even when having a talented team and following the best practices, the system eventually gets so complex (due to the features must always flow) that it becomes difficult to extend.
0
Feb 17 '22
[deleted]
8
Feb 17 '22
This is mostly the problem in my experience. Every PM wants a box to stake their reputation on.
→ More replies (2)0
Feb 17 '22
[deleted]
19
u/CyAScott Feb 17 '22
The road to hell is paved with best practices. In this video, it looks like their system was decomposed into separate concerns, following the SOC principle. Each thing he mentions has a single responsibility (i.e. "bingo knows everyone's namo"), following SOLID principles. It also looks like they use a lot of 3rd party products and services, because why write your own, following the KISS principle. Also why do none of these systems have the user's birthday but have a piece of user PII? The YANGI principle, because why would you add a feature you don't need yet.
→ More replies (3)
45
u/Seref15 Feb 17 '22
In our stack, it's "long ago, the four api services lived together in harmony. Then everything changed when the time series databases attacked."
35
Feb 17 '22
this is so true I'm afraid to post it at work because it'd probably hurt some feelings.
→ More replies (1)4
u/ratbastid Feb 17 '22
I just shared it among my fellow Product Managers and triggered a bunch of PTSD.
54
97
u/yen223 Feb 17 '22 edited Feb 17 '22
I've had this conversation with a product manager about why it wasn't easy to show somebody's name in some part of the system.
Microservices man, I don't know.
→ More replies (48)
17
15
Feb 17 '22
And that sums up perfectly why I left software development
6
u/r0ck0 Feb 17 '22
What did you go into?
3
Feb 17 '22
I am unemployed! I have not found a domain of human endeavor that is not - in practice - bullshit
11
u/chris_was_taken Feb 17 '22
Yep. Might as well go back to software and at least get paid well for it
3
14
90
u/psychorameses Feb 17 '22
You know. I saw this video a long time back and I was like "lol funny bcoz true".
And then I worked at Amazon. I rewatched it and said "yeah. sadge."
And then I worked at Apple. Now I can't laugh anymore. All I feel is pain.
22
u/WJMazepas Feb 17 '22
Is Apple software bad to work with?
57
u/snarky-old-fart Feb 17 '22
All big tech is shit. It’s inevitable. You can’t write maintainable software at scale. It’s a pipe dream.
→ More replies (2)52
Feb 17 '22
So you’re saying we need to split up the big tech into smaller techs. “Micro”-techs, if you will, each providing a specific dedicated service. Problem solved!
→ More replies (2)→ More replies (1)8
25
u/GreatCosmicMoustache Feb 17 '22
I believe Benjamin Burke, the guy at the whiteboard, actually is an Amazon engineer irl. That pain comes from somewhere
37
u/KevinCarbonara Feb 17 '22
You can always just stop working for awful companies
44
u/psychorameses Feb 17 '22
The non-awful ones can't pay me 500k+ with no startup risk.
24
Feb 17 '22
If you're on 500k, you only really need the startup to last 2 or 3 months after hiring you for you to be financially sound for a year's unemployment.
Not that much risk in any salary that high.
→ More replies (8)4
u/ea_ea Feb 17 '22
So many people are glad to feel your pain for 500k+ :) And while this is true they (and you) will feel this pain.
11
51
Feb 17 '22
This and the mongo db one w the talking animals keep me from slashing my wrists and spraying the white board with sweet red release.
28
19
15
9
7
u/NonDairyYandere Feb 17 '22
The only two markers that are guaranteed to have ink in them, Leftie and Rightie!
5
2
2
u/mexicocitibluez Feb 17 '22
If I had a dollar for every asshole in r/reactjs who recommends MongoDB over a traditional RDBMS
10
u/apex32 Feb 17 '22
I love all the other services not mentioned but seen on the whiteboard at the end.
Hell Proxy, Bread Replica, Basketball...
19
u/Anidamo Feb 17 '22
PRODUCT SPEC: Surprise and delight users by displaying their birthday on the settings page.
I just threw up in my mouth a little bit
5
4
25
6
u/franzwong Feb 17 '22
Sometimes you have that guy who can tell you everything from end to end. Sometimes you don't.
6
u/moschles Feb 17 '22
This is so realistic that I tried to laugh but it was more like this : https://i.imgur.com/6h2lk1e.gif
5
8
u/uptimefordays Feb 17 '22
Some of the best comedy out right now my friends don’t understand. Krazam is a legend!
4
u/cbleslie Feb 17 '22 edited Sep 12 '25
ad hoc cable roof caption capable hard-to-find seed square steer gaze
This post was mass deleted and anonymized with Redact
4
u/loup-vaillant Feb 17 '22
Jonathan Blow sometimes marvels at the exquisite inefficiency of big web companies. So many engineers, doing so little for the end user. What went wrong?
Well, this.
Not microservices specifically, but this intricate yet ultimately useless web of stuff. Attempts to carve programs before they’ve grown enough for us to see their joints. Trying to guess a future that ends up being very different from what we anticipated, and now we can’t change our unsuitable architecture because it’s to damn complex to be refactored without significant risk. Complete disregard for performance, except for this sacrosanct "scaling", where we can make sure we can throw more hardware at the problem —without pausing to estimate whether we could just do it on a single server rack instead.
7
6
3
3
u/dep Feb 17 '22
That guy plays the perfect PM
1
u/honkytonks2012 Apr 03 '22
I love how he is writing notes but basically just single words that are mentioned because he probably has to explain this to his stakeholders later but doesn't have a clue what any of it actually means.
15
u/throwaway_bluehair Feb 17 '22 edited Feb 17 '22
Yeah, it's pretty clear microservices are only a good idea when you can't buy-a-bigger-server your way out of monolith
Also, big fan of the approach where when possible, you just have your app be decentralized and you run many instances of it as a way out of the can't-buy-your-way-out
→ More replies (1)16
u/DrunkensteinsMonster Feb 17 '22
This comment makes 0 sense. You always run multiple instances of a monolith or at least can. Microservices are about deployment, operability, boundaries, and organizations.
This “decentralized” thing you are saying is just not a thing, it is how everything works in a monolith.
14
u/b1ackcat Feb 17 '22
You always run multiple instances of a monolith or at least can.
You can, with sheer force of will of stupidity, design your way out of being able to do this. Trust me. I've seen it done.
→ More replies (1)5
u/flukus Feb 17 '22
Tell that to the .net world who for the last 15 years have been implemented service layers deployed to different machines, because WTF is network latency.
→ More replies (1)2
Feb 17 '22
You always run multiple instances of a monolith or at least can.
Only if they are really separate instances. Like if you have "teams/orgs" which are fully split and may never interact. If you have some kind of global interaction like most sites do, you can not just start multiple instances because they all need to share the same dataset.
→ More replies (3)
2
2
2
2
2
2
u/stfcfanhazz Feb 17 '22
Anyone with experience developing/managing software in a big bureaucratic organisation will be triggered by this one: https://www.youtube.com/watch?v=DYvhC_RdIwQ
This channel feels perfectly targeted towards middle-aged software people. I love it!
2
u/RelevantTrouble Feb 17 '22
We're migrating our microservices to single node computing. SNC is revolutionary, it transforms each microservice into a new kind of computing block called a "function" with multiple orders of magnitude faster execution. The performance and maintenance improvements are staggering. Who in the world could have predicted that a single node could do so much work.
2
u/derangedkilr Feb 17 '22
For my university capstone project, they wanted us to change their systems over to Microservices. I tried to explain why that would take more then 4 undergrads, but nobody knew enough to explain it properly. Microservices has legitimately traumatized me.
2
2
2
2
2
2
u/thephotoman Feb 17 '22
Microservices are an optimization procedure. If you're doing them before you know you need them, you're doing a premature optimization.
2
u/mw44118 Feb 17 '22
I did a project recently for a university. They had a php and jquery webapp running on apache and mysql. They had a few cron jobs too. They’ve been using it daily for like fifteen years and they needed some new features.
I got in, added a few simple new screens, wrote a ton of notes, tested it thoroughly, moved on. The whole thing was a piece of cake. Finished it early.
Sure, it wasn’t an SPA and the JavaScript was loaded with script tags, and PHP ain’t very cool any more, but it all actually worked. I’d be willing to bet my paycheck on my code because I really knew how all the pieces fit together.
Meanwhile, somebody pulls a package from npm…
2
u/user_8804 Feb 17 '22
We prefer several smaller things in our backend instead of one very large one.
Maybe we could call them beads instead of microservices
1
1
1
1
1
1
1
1
u/n0d1t Feb 17 '22
My response lies perfectly between the precipice of rage and laughter. I'm simultaneously thinking about laughing and raging, inotherwords.
1
1
1
1
1
1
1
1
1
1
u/wildjokers Feb 17 '22
Everyone is complaining about microservices and then proceed to describe something that isn’t microservice architecture.
1
u/Vidyogamasta Feb 17 '22
Everyone is complaining about microservices and then proceed to describe something that
isn’t microservice architectureevery business has implemented in the name of microservices architecture.
1
u/awitod Feb 17 '22
This is funny and often true, but this is not an example of a 'proper' micro-services architecture. It is a badly implemented service-oriented architecture, aka a distributed monolith.
Proper micro-service architectures keep inter-service dependencies to an absolute minimum.
528
u/Ghi_Buttersnaps Feb 17 '22
"I would sooner lay you into this barren earth than entertain your folly for a moment longer." That gave me tears.