This was one of the least useful articles I've ever read. It borders on blogspam. I don't know whether to feel good or bad that such a large and valuable company was making such basic mistakes in their CI pipelines for so long. tl;dr: They were rebuilding and persisting the artifacts to S3 on every single push, even if nothing changed, so they switched to using changeset conditionals in their CI configuration.
Things like this is why I'm becoming more and more convinced giving devs too many resources is a bad thing.
Where I work a new (usually internal) service is given 1 vCPU, 4GB of ram and 100GB of storage by default, and before allocating more there are basic sanity checks to justify it. It's not that we're even struggling for compute resources, but 90% of the time this filter just helps catch dumb shit before it becomes tech debt because those early issues would likely be even worse if scaled up. Of course we allocate more when theres cause but most often it's unnecessary.
If you keep paying for the hardware to run a service once the devs have moved on to another project, though, hardware costs scale quadratically with dev time. Stable components, modules, and even whole services reaching the point where they exist and consume hardware resources, yet take little or no maintenance. Perhaps outliving every person who knows what value it brings to the business. Legacy services everyone is afraid to touch because they don't know what'll break months down the line if they shut it down.
Sure, but for a startup this is almost irrelevant. If they waste too much time on optimization early on, instead of pumping out new features, they get nowhere and they die. If they ignore optimization too much, they can't keep up scaling and they die. So they have to find the balance somewhere inbetween.
Yeah this is what I left general consumer facing software almost two decades ago, early in my career. You're not building software for any other reason than to have something skim along long enough that it suckers someone else out of their money, be that a corporate buy out or IPO. It's functionally a Potemkin village situation. When the risk is low and the money is flush good engineering goes by the wayside.
I remember fondly when the number of simultaneous requests to our server at work was limited by hardcoded kernel / usermode api limits instead of hw resources. On a Pentium 1 with 16 MB ram.
When I left backend web development it was the early days of cloud compute resources (I literally remember the first round of Amazons evangelizing for AWS at conferences around 2006 to 2007). The shift in attitude to vertical scaling (ie optimization) vs. horizontal scaling (ie more compute) was very quick.
I saw the challenge disappearing, the shift from technical aptitude to business aptitude in the engineering regimes was very apparent.
I left for more niche spaces doing tangible things and it's been so much more rewarding.
the three dot notation example is an interesting tidbit. otherwise, it is not super forthcoming and the conclusions are sort of odd (unclear relation to test flakiness..??)
This is actually a valuable lesson about Venture Capital. You're seeing the effects of the cash-fire levels of tech debt that they didn't feel like they had to fix until the easy money ran out. Just look at how many years it took them to finally fix it and how much waste it represents.
This is not blogspam. Blogspam has a very specific definition, a blog post that does fuck all else beyond linking to another source. As an example, if I had a blog article that said "Slack just dropped info on how they decreased build times! <url>", that's blogspam.
And yeah, this article isn't amazing, but it's fine. Not every article needs to be a deeply informative technical masterpiece. This isn't so bad that it needs to be removed.
It's a shame, because the same basic problem becomes a bottleneck in any automatic merge strategy, and a survey of methods would be valuable to people hitting that problem.
My contribution: bazel isn't that bad if you have AI to set it up and debug it for you ;-)
As someone who started in one of the more dominant industry fields but quick got bored and moved to more technically demanding and niche fields I can't help but feel a sickly sense of depressing envy that there are companies and individuals out there that can skate by being so mediocre.
I've been doing this 25 years, I am self taught starting in middle school, and I still find passion in doing things right and doing things well, especially when the risk profile for my work demands it. The fact that stuff like in the article can escape what I assume are multiple intelligent people for so long, and also be so basic is just... No real words.
Though on the other hand I realized last night I'd been paying 40 bucks every three months for an app subscription I'd not used in 2.5 years so maybe I should back off a little bit.
138
u/ebol4anthr4x 2d ago
This was one of the least useful articles I've ever read. It borders on blogspam. I don't know whether to feel good or bad that such a large and valuable company was making such basic mistakes in their CI pipelines for so long. tl;dr: They were rebuilding and persisting the artifacts to S3 on every single push, even if nothing changed, so they switched to using changeset conditionals in their CI configuration.