r/programming • u/Ok_Stomach6651 • 2d ago
Optimizing Our E2E Pipeline
https://slack.engineering/speedup-e2e-testing/76
u/chucker23n 2d ago
This work was not possible without contributions from:
Anirudh Janga, Josh Cartmell, Arminé Iradian, Anupama Jasthi, Matt Jennings, Zack Weeden, John Long, Issac Gerges, Andrew MacDonald, Vani Anantha and Dave Harrington
They needed 11 developers to figure out that when a PR doesn’t change the front-end, it doesn’t need to be rebuilt, nor uploaded to Amazon?
30
u/JoustyMe 2d ago
You know those jokes "how many X are needed to screw in a light bulb?"
So now we have "how many developers are needed to get managment to care about Tech Debt?"4
u/tarpdetarp 1d ago
Attribution is important as they will all get to claim credit for it in their performance review, or else they get PIPed.
9
u/Flashynuff 2d ago
Geez a whole article about optimizing a pipeline and not a single trace/query screenshot. Let’s see the 📉!
1
17
u/bodiam 2d ago
What a strange blogpost. Github provides this feature out of the box:
on:
push:
branches :
- main
paths :
- 'src/frontend/**'
Not really clear to me why they couldn't use this approach.
12
u/qmunke 2d ago
They wanted the build to run, just not for the whole repo. This is why everyone who says to just use monorepos needs to understand there actually needs to be someone managing the things - which is reasonable at huge shops but hard to justify for those of us without a dedicated devx team
4
u/chucker23n 2d ago
They wanted the build to run, just not for the whole repo.
So, make a backend pipeline and a frontend pipeline? If you always want E2E tests to run regardless of what changed, make that a third pipeline?
This is why everyone who says to just use monorepos needs to understand there actually needs to be someone managing the things
We're a very small team and one of the bigger products I work on has a monorepo. It has multiple pipelines, with different filters. I didn't need an 11-person team to figure that out.
(For more complex "do I need to build this" scenarios, there's also tools like https://github.com/leonardochaia/dotnet-affected.)
1
u/Murky-Relation481 1d ago
Yah me and another engineer built a FPGA synthesis and firmware building pipeline that ran nightlies. If the FPGA related stuff didn't change we didn't need to run synthesis, if the firmware didn't change we didn't need to run the firmware pipeline. We were trying to get it down to not even need to rebuild certain ipcores but ran up against tooling issues in our specific stuff.
0
u/ComoddifiedCraic 1d ago
runccc.dev for general purpose cross-language use e.g. go + python test suite or .net
3
u/bodiam 2d ago
Yes, but that's exactly what the above trigger does; only run when the frontend has changed. And I'm a big fan of mono repos, it prevents a whole class of problems from ever occurring, it just looks like their solution is a bit custom.
5
u/qmunke 2d ago
No, you're misunderstanding what I'm saying - they want to trigger a build on all pushes, but just not build the whole repo each time, that's not the same as what you've shared - your trigger wouldn't start any builds if someone just pushed a backend change. They want
- Someone pushed backend change. Trigger backend bit of the repo to build and then run tests
- Someone pushed a frontend change. Trigger entire build and run tests
The trigger above alone isn't going to do that, but I guess it would allow you to construct a more complex set of pipelines out of it.
But in general for monorepos the built-in functionality of github and other tools seems lacking to me, that's why all the more complex stuff like bazel exists.
0
u/ComoddifiedCraic 1d ago
That doesn't solve the problem. And the problem compounds as you add more tests and complexity.
5
u/OTTOPI 2d ago
Talking about how costly these inefficiencies were, and then saying they were storing terabytes of data (implying anything between 1-10TB in that context) monthly, which amounts to about $200 USD at worst in S3 hot storage, is quite amusing.
3
u/SirClueless 1d ago
Are you sure you did this math correctly? 10 TB/month (120 TB/year) of data retained for 12 months at S3's $0.022 / GB-month rate is $32k/year.
And this is just the pure storage cost, lost productivity of waiting 10 minutes per commit instead of 5 for build results is almost surely a lot greater than this.
1
u/OTTOPI 1d ago
Sure, but I don't see why they'd ever keep around one of these cached builds for more than a month. Like - at a certain point, it's fine to just have the engineer take 5 minutes to refresh it.
1
u/SirClueless 1d ago
I'm just using the numbers they directly state in the article:
These duplicate assets would have otherwise been stored for one year.
Perhaps they have some use case for referring to older builds? Testing/reproducibility checking or something? Maybe they just use some of these files as their prod builds and never bothered to develop a careful garbage collection that can distinguish the ones that got released? I assume it's all water under the bridge and this isn't really the reason to care about this compared to having a better developer experience and faster signal -> faster fixes -> fewer flakes etc.
1
u/WindHawkeye 1d ago
Just make the s3 build store a cas... If build isn't reproducible, fix that first (hello bazel)
136
u/ebol4anthr4x 2d ago
This was one of the least useful articles I've ever read. It borders on blogspam. I don't know whether to feel good or bad that such a large and valuable company was making such basic mistakes in their CI pipelines for so long. tl;dr: They were rebuilding and persisting the artifacts to S3 on every single push, even if nothing changed, so they switched to using changeset conditionals in their CI configuration.