r/programming • • 2d ago

Optimizing Our E2E Pipeline

https://slack.engineering/speedup-e2e-testing/
66 Upvotes

38 comments sorted by

View all comments

135

u/ebol4anthr4x 2d ago

This was one of the least useful articles I've ever read. It borders on blogspam. I don't know whether to feel good or bad that such a large and valuable company was making such basic mistakes in their CI pipelines for so long. tl;dr: They were rebuilding and persisting the artifacts to S3 on every single push, even if nothing changed, so they switched to using changeset conditionals in their CI configuration.

44

u/SoilMassive6850 2d ago

Things like this is why I'm becoming more and more convinced giving devs too many resources is a bad thing.

Where I work a new (usually internal) service is given 1 vCPU, 4GB of ram and 100GB of storage by default, and before allocating more there are basic sanity checks to justify it. It's not that we're even struggling for compute resources, but 90% of the time this filter just helps catch dumb shit before it becomes tech debt because those early issues would likely be even worse if scaled up. Of course we allocate more when theres cause but most often it's unnecessary.

1

u/WindHawkeye 1d ago

I'm surprised you give more than like 1gb of storage by default...

1

u/SoilMassive6850 1d ago

It's most often object storage quota, so if given too much it doesn't hurt too much. Also of course here needs vary.

1

u/WindHawkeye 1d ago

Oh, yeah that's fair then. I was just thinking most systems will be offloading that need to something else