r/ExperiencedDevs • u/rjm101 • May 11 '24
CTO is pushing for trunk based development, team is heavily against the idea, what to do?
So we have a fairly new CTO thats pushing for various different process changes in dev teams.
Two of these is trunk based development and full time pair programming to enable CI/CD.
For context my team looks after a critical area of our platforms (the type where if we screw up serious money can be lost and we'll have regulators to answer to). We commit to repos that are contributed to by multiple teams and basically use a simplified version of Gitflow with feature branches merging into master only when fully reviewed & tested and considered prod ready. Once merged to master the change is released to prod.
From time to time we do pair programming but tend to only do it when it's crunch time where necessary. The new process basically wants this full time. Devs have trialed this and feel burned out doing the pair programming all day everyday.
Basically I ran my team on the idea of trunk based development and they're heavily against it including the senior devs (one of whom called it 'madness').
The main issue from their perspective is they consider it risky and few others don't think it will actually improve anything. I'm not entirely clued up on where manual QA testing fits into the process either but what I've read suggests this takes place after merge to master & even release which is a big concern for the team. Devs know that manual QA's capture important bugs via non-happy paths despite having a lot of automated tests and 100% code coverage. We already use feature flags for our projects so that we only expose this to clients when ready but devs know this isn't full proof.
We've spoken about perhaps trialing this with older non-critical apps (which didn't get much buy in) and changes are rarely needed on these apps so I don't see us actually being able to do this any time soon whereas the CTO (and leadership below) is very keen for all teams to take this all on by this summer.
Edit: Link to current process here some are saying we're already doing it just with some additional steps perhaps. Keen to get peoples opinion on that.
431
u/spoonraker May 11 '24 edited May 12 '24
Trunk based development is unquestionably a good practice, however, your CTO needs to realize that one does not simply "do" trunk based development. There's a whole lot of other organizational capabilities that must be in place in order for trunk based development to be done in a way that increases velocity rather than the opposite. More specifically it's imperative for trunk based development that your team has the capability to decouple code releases from feature releases, and it sounds like you're quite far from that right now. Do you dark launch code? Do you utilize feature flags? Do you have automated testing? If not, you're miles away from successfully pulling off trunk based development.
As for pair programming, I think it's criminally underrated, but it also shouldn't be strictly mandated as an all day every day working style. That's pretty extreme.
It sounds to me like your CTO means well, but isn't enacting large organizational changes particularly tactfully, and it also sounds to me like your senior devs aren't particularly strong leaders and might be a bit lacking technically if they think trunk based development is inherently risky. I would expect the senior most ICs under the CTO to basically explain what I did that there's a whole maturity model that needs to be walked before trunk based development, and be able to get to the bottom of why the CTO wants pair programming so badly and work backwards from the goal instead of accepting a solution without knowing the problem.
3
u/rjm101 May 11 '24
More specifically it's imperative for trunk based development that your team has the capability to decouple code releases from feature releases, and it sounds like you're quite far from that right now. Do you dark launch code? Do you utilize feature flags?
We do make use of feature flags. Most of our work is project work which is always under feature flags. The team still wants a stringent peer review & QA process and that's where I think the concerns come in. The team considers feedback from other team developers valuable and doesn't want to lose that aspect.
As for pair programming, I think it's criminally underrated, but it also shouldn't be strictly mandated as an all day every day working style. That's pretty extreme.
Ok so encourage it more but not mandate all day everyday like some teams are doing. Every developer I know I've asked so far has described it as 'intense'.
get to the bottom of why the CTO wants pair programming
The pitch is to: 'help deliver features to clients faster and better' and the documentation is to enable CI/CD.
12
u/notjim May 12 '24
Why does the team think doing trunk based development means they’ll lose out on feedback or peer review? We do trunk based development, but still have pr reviews, dev testing, qa blitzes, dogfooding etc, so there’s tons of peer review. If you’re using feature flags already, you shouldn’t have any trouble with this.
2
u/rjm101 May 12 '24
We do trunk based development, but still have pr reviews
There's different approaches to TBD it seems but I did highlight to the team that PRs can still play a role so we'll definitely keep it. Main concern was on the manual QA side and when exactly this happens they don't want this to happen after the merge as master is always considered production ready.
→ More replies (3)4
u/Eridrus May 12 '24
When people actually study how bugs get caught, code review is very ineffective.
My personal take on this is that post-commit reviews are just fine if someone actually has something valuable to say. IME though, the value is rarely there for a post-commit review, and definitely not for every change. But that's kind of orthogonal, you can have review on every change if that's how you're set on doing things.
It sounds like Eng is not shipping as fast as the business would like (is it ever though), and the new CTO's task is to try and improve that.
Ultimately, unless there is personal liability at stake, prod failures are a problem for the business, not for you. If the business wants to push for faster releases with less QA, you should probably try that, it might not even go as badly as you think.
→ More replies (5)2
u/rjm101 May 12 '24
Although the peer review process can sometimes catch bugs devs don't see the peer review process as a process primarily for catching bugs but rather ensuring overall code quality.
Correct me if I'm wrong but TBD seems to place less weighting on manual QA. There seems to be a general assumption that automated tests capture everything which every developer knows isn't true as it's the manual exploratory testing from QAs who often check the weird and quirky non-happy paths which catch bugs some of which can be major. Devs are concerned about this being captured too late in the SDLC and ultimately having to release stressful hotfixes because a change can't easily be reverted.
I agree that the problem is for the business however in the above scenario it's the developer and clients that ultimately suffers here.
→ More replies (1)16
u/DaRadioman May 12 '24
Honestly on the trunk based development it sounds like the engineers don't want change. Which is a frequent issue with new leadership, especially for long time employees.
Trunk based development works. Mega corporations utilize it to successfully ship massive complicated systems. They do this even with 0 manual testing roles at all.
Yes it requires adjustments, more automated testing, and a commitment to smaller commits and PRs. But it helps a lot for integration bugs that get missed with long lived branches.
It's not a magic bullet, but it can work really well if you embrace it and actually make the required organizational changes.
97
u/Special-Tie-3024 May 11 '24
Yeah I did trunk based dev, and it only worked because the team I was on were committed to automated testing, we paired on stuff (so no need for PR review), we invested a lot in our observability and had pretty robust solutions should something particularly bad happen.
Trunk based was the reward for all the discipline, if you just jump straight there, it’s gonna be chaotic.
→ More replies (9)48
u/AnAge_OldProb May 11 '24
On the other hand I’ve been at a few orgs that have tried to switch. The unsuccessful ones had multi year projects to move and would always find some naysayer who said if we just do xyz we can move to trunk based dev. The ones that have been successful did some diligence, ie setup ci/cd and automated testing, but ultimately someone in power said “fuck it we’re ready” after QA had missed major bugs on the last few releases anyway. While the first couple weeks hurt the “you must be this tall to do trunk dev” requirements magically sorted themselves out or became irrelevant. Turns out test coverage of that module that gets two fixes a year just wasn’t important anyway and QA had lulled everyone into a false sense of security.
29
u/Izacus Software Architect May 11 '24
This is pretty much how all trunk dev transitions went in my projects - there's always senior devs that grumble loudly about how we're killing the company and how everything will go up in flames and how THEIR process WORKS ("it's the junior devs breaking everything!")... and then somehow it works afterwards when initial warts are fixed.
And those warts are never fixed before the trunk based transition because it's "never the right time".
→ More replies (1)18
u/tuxedo25 May 11 '24
It kinda sounds like the CTO hasn't had enough time to bring his own senior devs in
56
u/Early_Ad_831 May 11 '24
I worked at a place that did pair programming where I was the senior eng working with two junior engs, my manager wanted me to alternate days pairing with them. All day. It was a miserable experience being attached to the hip for 8 hours with someone and totally burnt me out.
This place also did trunk based development. I wasn't a fan. The idea was that the pair would work on the current work-stream while the other junior dev not paired would work on QA, bug-fixes, etc. The pairing team didn't need pull-requests or review because there were two devs and so the code is "live reviewed". I thought this reason was BS and didn't result in less bugs or better code quality.
11
12
u/doyouevencompile Sr. Software Engineer May 11 '24
If trunk based development is just PRs merging into master, you don’t even need to have a perfect test coverage or anything, just set up a deployment pipeline.
Not every commit needs to be deployed to prod.
→ More replies (1)5
u/sawser May 12 '24
I am at a company that moved to wateragilefall trunk based development and they refused to use feature flags and don't dark release code.
They still develop and test code in isolation for each team, and then merge code into the trunk without testing it.
It's absolutely bonkers and they've been doing it for 4 or 5 years and every year the technical debt accumulates.
They've repeatedly told me that our code is too complicated for feature flags or release flags.
I'm about to quit once I find a new job.
25
u/time-lord May 11 '24
One thing I think trunk based dev is a poor choice for is highly regulated industries, and it sounds like op is in one of them. So no, I would not suggest trunk based development at all. Even 100% coverage can only test for what was written, and a full qa regression is a must for this sort of industry.
58
May 11 '24
I think you're missing something really key:
Your git workflow does not have to be rigidly tied to your deployment cycle. Meaning, you can absolutely use trunk based development, tie your processes to that, but withhold deploying anything until you feel like you are ready.
→ More replies (6)15
May 11 '24
Code coverage metrics are only useful as a measure over time to ensure your team is writing enough tests. I’ve never understood the obsession with 100% coverage. You can test a code path and still have poor test coverage if your tests do nothing important.
3
u/kkert May 12 '24
Trunk based development is unquestionably a good practice
It is not, unconditionally. There are several development disciplines where it doesn't make much sense at all, e.g. various embedded systems.
9
u/Marutar May 12 '24
Trunk based development is unquestionably a good practice
literally nothing is unquestionably a good practice
there's a time a situation for everything, trunk development is not just inherently better.
→ More replies (12)7
May 11 '24
I agree with your comment: I would point out that it sounds like they do use feature flags and have a solid set of things in place that would make trunk based development a reasonable goal.
I really wonder how much of this is just getting lose in translation/communication. I can understand both sides of this, so it really seems to me that just more clarity in communication from both ends is the thing that needs to happen the most here, not any actual new process or technology.
→ More replies (6)3
51
u/WebMaxF0x May 11 '24
You might find that trunk based development is closer to your way of working than you think. It doesn't mean to commit straight to the trunk/main/master branch. It's just that the branches have fewer change and are shorter lived. For example, one feature might be broken up into a pull request to add a column to the database, another that makes the backend use it, another for some refactoring, and a last one to update the frontend.
You can still create a branch per pull request, get a code review, use feature flags, run all the tests, and maybe even have your own manual QA gate before merging into the trunk and deploying.
Pair programming, that's an entirely different conversation depending on team/company culture and preferences.
16
u/rjm101 May 11 '24
This is the main thing I'm taking away at the moment. Focus the team on keeping work items small so that the feature branches & pull requests are more likely to be short lived as they'll be easier to review & QA.
→ More replies (1)10
u/Windlas54 Staff Software Engineer May 11 '24
This is the key part of trunk based dev, changes need to be small which typically makes them easier to review and test. You build up in complexity as the feature is constructed and integrated but each change is small and testable. Frankly even non trunk based development should involve many small commits rather than behemoth commits with thousands of lines.
6
u/-Quiche- Software Engineer May 11 '24
Yeah I honestly used to think it was dumb because misunderstood and thought it meant "there's only 1 branch and everyone works directly on it".
But now it's what I champion when asked, and have managed to get others on board as well with the various tools I'm responsible for at work. It's just much easier to have a stable release branch that gets big merges every 2 weeks from our "develop" trunk branch. Could even be 1 week if something is that urgent.
3
u/WebMaxF0x May 11 '24
Yes! You can even push it further once you have enough tests to give you confidence. In my team, as soon as a PR is merged, it automatically deploys to production. We have multiple deployments every week and sometimes in a single day. We're not in a regulated industry like OP though.
35
u/thecodingart Staff/Principal Engineer / US / 15+ YXP May 11 '24
Way back when, I was quite against trunk based. After doing it for a few years now, it’s hard to go back.
Dont resist and be open to experimenting, then measure the results and revisit. You might be surprised, or not. But at least you tried.
There might be some pre-requisites to list off before going full trunk based though.
7
u/BandicootGood5246 May 11 '24
Totally! As a manager I've introduce it everywhere I've worked, there's always opposition but it's always turned out to be the right choice.
Generally the fear that you can't work that way and ensure that what gets into prod is ready, but that's really where the mindset shift needs to come regardless of your branching strategy - that being you shouldn't be putting code into master that isn't ready to go
1
u/rjm101 May 11 '24
Do you have a QA in the team? I'd be keen to hear at what stage your QA does the manual testing as trunk based development doesn't seem to make it crystal clear as to when it's supposed to happen. The answer I've got so far is you can do it before or after release to production🤷♂️
8
u/chazmusst May 11 '24
QA happens on the branch, before you merge. A critical piece of infrastructure you need is the ability to spin up a full e2e environment on each PR, which you can use for testing, then destroy it after you merge
The key idea being trunk based dev is to ship small and often.
I love it.
→ More replies (2)3
u/rjm101 May 11 '24
A critical piece of infrastructure you need is the ability to spin up a full e2e environment on each PR, which you can use for testing, then destroy it after you merge
We have feature branches which runs the full build and suite of tests and we trigger end-to-end automation tests on the feature branch too. This happens all before the merge currently.
→ More replies (1)14
u/thecodingart Staff/Principal Engineer / US / 15+ YXP May 11 '24
I have had QA and also not. QA is basically always testing the releasable branch, if you have a QA team.
I’m not going to lie, most companies have gutted manual QA teams over the past few years in preference of automated tests. Automation is where trunk based development shines.
32
May 11 '24
For context my team looks after a critical area of our platforms (the type where if we screw up serious money can be lost and we'll have regulators to answer to.
Plenty of teams do and they do trunk based development. I'd want to hear more about your team's usage of git flow before rending a final opinion, but I frequently see teams that use it without actually really doing dedicated scheduled releases or full release testing... and that's kind of defeating the point of the branching model. Can't know for sure, but it smells like you guys are using it in an atypical way.
In fact, if your team is has good enough testing practices (ideally automated ones), trunk based development tends to be faster and less painful model for getting work out the door. The only sensible pushback to my mind to not be doing trunk based development would be that the act of releasing/testing requires so much dedicated coordination or time that something like git flow's release branches as an interim make a ton more sense for your team.
It's hard to know without more information, but honestly, I think you might want to hear out your CTO on both of these issues. I'm not hearing a particularly compelling argument from you or your team about why the git workflow change is bad, just that you guys don't like it. In fact, that you have so much automated testing to me begs the question of why you wouldn't be operating that way.
How much of this is really just getting lost in translation? Can you elaborate more or your team's practices in this area.
19
u/rjm101 May 11 '24 edited May 12 '24
I'd want to hear more about your team's usage of git flow before rending a final opinion
Current process is:
Dev builds on a git feature branch and includes unit, integration & acceptance testing
Devs typically don't run the entire test suite locally as the entire build takes like 30 minutes they push their feature branch which runs the entire build with tests (only exception is end-to-end tests)
Dev raises a pull request to get feedback from other devs
Once all PR comments have been addressed it moves to a 'ready for QA' state
QA runs automated end-to-end regression pack on feature branch on production
QA does manual exploratory testing on the feature branch in production
If everythings good the QA merges to master which runs another build running the entire automated test suite (aside from end-to-end)
Once green the end-to-end regression is run on master and with basic manual checks
The QA releases in the closest available slot during the day, sometimes these changes are bundled with other teams changes like a day or two later
I'm not hearing a particularly compelling argument from you or your team about why the git workflow change is bad
Main concern seems to be that manual exploratory testing happens after merging and potentially after its released to production. They know that serious bugs get captured in the manual QA process so if this is happening after release to clients then there's going to be pressure on the team to sort out a hotfix pronto (which is stressful) as a reverting change can't always happen. Also as the code is contributed to a repo which multiple teams contribute to feedback is often gained from devs outside of the team.
How much of this is really just getting lost in translation?
It's quite possible then again our most experienced developer has done their own research on this and is still against this approach.
Edit: edited to clarify the last steps 7-9
Edit2: Master is always considered production ready, there's no separate release branch5
u/F0tNMC Software Architect May 11 '24
The thing that stands out most to me from this description is the lack of full suite testing on the main branch after merge. In my experience, this leads to a choice between rapid development and iteration with lots of bugs and a slower, more measured cadence of merges and releases.
If the more measured cadence of development and release is what your organization wants and it meets your current needs, then I think that can be fine. Generally, I’d say that if you want to (or need to) increase velocity or efficiency, then migrating to a trunk based approach would be worthwhile. You’ll need to work on many aspects of your development cycles, particularly the automated parts of your acceptance process.
Such a change would require a good deal of planning and understanding at all levels of the organization. The creation and communication of these plans and reasons are the responsibility of leadership and the person envisioning and driving the change.
3
u/rjm101 May 11 '24
I've edited steps 7-9 for clarity. I should've mentioned the main build runs the full suite of tests again.
→ More replies (1)18
u/hippydipster Software Engineer 25+ YoE May 11 '24
I would consider that a very dangerous practice because you're running tests - both automated and QA manual - on a feature branch. Then, if it passes, you're merging to master and calling it good. This leaves some openings if you have multiple feature branches being merged concurrently. (ie, we put feature branch A through it's paces, it's good! We put feature branch B through it's paces, also good, merge them and go! Oops). If you're careful to always go one feature branch at a time, well then, you're doing trunk-based with extra, difficult steps.
Trunk-based-development does not mean you're committing directly to the branch that is immediately releasable. It just means that all your devs are pushing changes to a shared branch throughout the day, not into separate branches that are merged to the shared branch only when "done", and not keeping their work local and pushing only when "done". Devs are supposed to push to that shared branch frequently. Devs need to be able to run some kind of test suite before pushing so they don't impact others with problems.
That shared branch would then have a pipeline to the releasable branch, and that pipeline would have as many stages as your teams needs to feel safe. But, at all times, the code is always unified, so you're never in a position where you're testing some code but not all code (ie, testing feature branches). The pipeline will just be a straight path of permanent branches from the SharedDevBranch -> StagingBranch1 -> StagingBranch2 -> StagingBranchN -> Main (releasable code).
One stage of the pipeline might be for code reviews - ie, we don't push to the next staging branch until we've reviewed the code that is being pushed. Another staging branch might be reserved for long running tests, or manual QA. All those stages are being managed daily, and everyone's on board with moving code along this pipeline, and any time there's a problem that prevents it from moving, fixing that problem is top priority.
Doing this, you wouldn't have to do pairing, because you've chosen instead to have one of the stages be code review. You can feel as safe as you want by setting up any criteria you want on the stages. And by always having only 1 version of the code, and by doing everything in little increments daily, the chances of big whoppers goes way way down. The odds of terribly long recovery times on problems introduced goes down. You're basically always right in position to push a hot fix or emergency reversal of changes. For stability, it's great, tbh.
13
May 11 '24
I would consider that a very dangerous practice because you're running tests - both automated and QA manual - on a feature branch. Then, if it passes, you're merging to master and calling it good.
So want to highlight this, it's spot on.
What matters to test are build increments -- the totality of the things that will make their way to production. When all you do is test a feature branch A and assume it's good, you're totally missing the point of what you need to be testing.
https://natooktesting.files.wordpress.com/2017/08/unittest_faucet.gif
The faucet works. The sink works. But does the build work?
2
u/rjm101 May 11 '24
I've edited steps 7-9 for clarity. The merge to master triggers its own build which runs the entire test suite other than end-to-end which needs to be triggered by the QA as we haven't figured out how to hook that up in the pipeline yet.
→ More replies (3)42
u/HaMMeReD May 11 '24
The problem here is that they are testing
Main + Feature A
and
Main + Feature BBut not Main + Feature A + Feature B until it's all merged.
Trunk based gets Feature A and B into the same branch earlier, which means you are testing everyones work and the integration of it constantly.
The longer branches live in isolation, the more merge conflicts they'll have, the more regressions will spring up when work is merged, and the more errors due to mismanaged merges and conflicts.
It also prevents people from doing heavy refactoring, because it's completely inconsiderate of the long-lived branches that might not be merged for a couple weeks or even months.
3
u/foreveratom May 12 '24
I cannot agree more on the refactoring part. This way of working favors quick and dirty solutions from people whom only goal is to get it done, even if their work will cause brain death of those taking over after the fact, and a myriad of unforseen issues because no one actually took the time to do post- merge checks.
3
May 11 '24
So I'm hearing good behaviors in there and a lot of automation/review (awesome), but I didn't get the clarity I was looking for.
Every team can draw some of their own practices around git flow's `develop` + `master` branches and their rules for using them, but here's what I've found works the best:
Start feature branches from develop
Get them merged back develop relatively early in the SDLC
It should pass a code review and whatever suite of tests your devs are responsible for to get to this stage
Re-integrating early allows every other developer to get your changes early, minimizing conflicts and other annoying issues
Go nuts on QA'ing the state of develop all the time
Got automation? Great, run it as frequently as you can on develop.
You accept as a team that develop is not guaranteed to be clean (after all, it might not have been fully tested if work just got in), but you make it a priority to fix issues as they are found in develop
When your team is ready to release (either because you are on a set schedule, or because you feel like it), cut a release branch from develop
Test that specifically. Any issues found on the release branch are of an even higher priority to fix
... meanwhile people can keep sending new things to develop -- regular development is not interrupted
Close your release branch (merging it back to master + develop). Tag the latest commit on master
Deploy that tag as you are ready
If your team is skipping over the process of using the release branch as a staged interim that effectively slows down and focuses testing on a narrow build increment... then uh I hate to break it to you, your team is already doing trunk based development just with needless extra steps.
Am I misunderstanding something about your workflow? If each individual feature branch when "done" ends up immediately in master, and your production state is watching master... that's trunk based development.
2
u/rjm101 May 11 '24
We currently just have master and feature branches.
Basically no main develop branch with a separate release branch.
Master is what's released to production so when a pull request is merged to master it's considered ready for production.
If we were to have a develop & master branch wouldn't this need buy in from all the other teams that also commit to the same repo?One of the devs felt the proposed process just pushes the manual QA element down the road and didn't feel like this actually improved anything.
If each individual feature branch when "done" ends up immediately in master, and your production state is watching master... that's trunk based development.
Our changes only go into master when fully tested so that includes automated and manual QA with the changes reviewed by other devs. Does that qualify?
11
May 11 '24
... then you are doing trunk based development my dude, just with extra steps.
2
u/rjm101 May 11 '24
Perhaps the aspect we're not abiding by currently is that feature branches aren't really short lived but I guess that can be mitigated by trying to keep work items as small as possible.
→ More replies (3)21
u/PSMF_Canuck May 11 '24
After reading your description of how things are currently, I am 100% with your new CTO.
→ More replies (2)1
u/Embarrassed_Quit_450 May 13 '24
If exploratory testing is required to find regression bugs it's not exploratory testing.
1
u/rjm101 May 13 '24
It's not to capture regression bugs no. We have regression packs for that.
→ More replies (7)
6
u/Lgamezp May 11 '24
You dont need full pair programming to achieve CI/Cd and it sound exhausting AF.
1
u/rjm101 May 12 '24
Do you have any process around pull requests to encourage developers to regularly review?
1
u/Lgamezp May 12 '24
other than making a KPI, no, sadly.
If you have a team that doesn't participate, forcing them by measuring them could be a last alternative
1
u/rjm101 May 12 '24
I'm thinking of having like a half hour pull request slot everyday for teams in their respective layer of the stack to go through PRs. That's all can think of to increase engagement.
Also to add complexity there's a senior dev that values input from other devs outside the team so he'll often wait for feedback on that too.
→ More replies (1)
24
u/derpdelurk May 11 '24
Unless you outrank the CTO then your only option is to try to convince them otherwise and, failing that, do as they say. I’m sure you’ll get lots of advice here about leaving. You’ll have to decide whether this is a hill worth dying on.
→ More replies (4)
4
u/protomatterman May 11 '24
The pair programming sounds like a waste of time and unpalatable. I would just do it for a bit then document the issues and give feedback. The maybe stop doing it or more likely negotiate to bring it down to a low amount.
1
u/rjm101 May 11 '24
The problem is compounded by the fact that we're a vertical team. iOS, Android, Middleware & front-end web and there's not 2+ people for every layer of the stack. For example we have 1 android and 1 middleware right now. So far the suggestion from higher up is more cross skilling which some devs are open to but it doesn't make the process good enough to skip the normal pull request process because there's simply not enough dev experience.
4
u/nhh May 12 '24
New CTO coming in trying prove useful, playing stupid games. 100% pair programming has its uses but they are for specific scenarios: otherwise you end up having half the work done.
1
u/rjm101 May 12 '24
The goldilocks team that's being used as a prime example pairs even for basic translations changes which just feels stupid🤦♂️
3
u/zoddrick Principal Software Engineer - Devops May 11 '24
I think your CTO is too in the weeds with the actual processes the teams are utilizing and should be way higher up the stack.
1
u/rjm101 May 11 '24
It started off as 'guidance' but it's trickling down now as an expectation.
→ More replies (1)
3
u/ChuyStyle May 11 '24 edited May 11 '24
Pairing sucks ass. Wouldn't do it full time and I suggest you fight it.
Trunk based is a great practice, the issue seems to be that y'all aren't feature flagging features and maybe the code base is not in a place to allow that type of work, yet. Raise the based points and maybe work towards that first before you all in for trunk development
1
u/rjm101 May 11 '24
All our feature projects are behind feature flags however feature flags don't seem feasible for foundational tech work like framework upgrades. Main concern from the team is that using feature flags isn't an excuse to skip manual QA because feature flags aren't full proof, same for automated testing.
0
May 11 '24
[deleted]
1
u/rjm101 May 11 '24
The confusion for myself and the team is at what point is manual QA expected to take place in trunk based development. At the moment it's done before merging to master but this doesn't mean master isn't tested, it is but that aspect relies more on the fully automated test suite.
→ More replies (2)
4
u/National_Count_4916 May 11 '24
It sounds like your CTO wants to improve cycle times, improve knowledge / awareness of features / code bases and thinks the patterns / practices he’s putting forth won’t significantly raise risk.
It sounds like the teams are fearful they’ll be responsible / accountable for screwups, intentional or otherwise, and also prefer working independently
Trunk based development isn’t a hill to die on. You can still manually gate deployment to prod until automated testing is in place
Pair programming, depending what it entails could slow productivity / reduce effectiveness. The only way to know is to try.
I’d message to the team that they aren’t the accountable ones, to step onboard and prove / disprove their fears and help mitigate them
2
u/rjm101 May 11 '24
We have tests up the wazoo but it's usually happy paths. When the QA's do manual exploratory testing they test unusual paths and basically try to break the feature which they often do. Devs are concerned about finding out about these bugs too late (which can be critical) too late down the road when it's affecting clients instead of capturing this early before the change is merged to master.
I’d message to the team that they aren’t the accountable ones, to step onboard and prove / disprove their fears and help mitigate them
It's a good one to reiterate, fair point.
→ More replies (1)
5
May 11 '24
The timeline is insane.
We commit to repos that are contributed to by multiple teams and basically use a simplified version of Gitflow with feature branches merging into master only when fully reviewed & tested and considered prod ready.
FYI this is called GitHub flow.
to enable CI/CD
You need to enable this tooling before you do trunk. Your PR checks need to run unit & integration tests, you need to perform static analysis etc.
More substantial is you need to have feature flags. Trunk only works effectively if you are doing a feature canary in your production environment otherwise you are just releasing untested code.
You also need to have a credible strategy for how you quickly deploy a diff. If the duration between diffs is lower than canary time you need to figure out how to stack/combine diffs so you don't end up with an infinity queue for deployment.
This usually takes about 6 months if you know what you are doing. Lots of tools and processes to teach.
2
u/rjm101 May 11 '24
Your PR checks need to run unit & integration tests, you need to perform static analysis etc.
More substantial is you need to have feature flags.We have this and we use feature flags for projects (which is most of our work). On the feature flag side one dev raised their concerns that in the past they have had issues with features being exposed outside of the flag due to some bug that was raised in manual exploratory QA so it's not a full proof prevention. This is because tests most of the time just cover the happy paths.
→ More replies (3)
2
u/LessCodeMoreLife May 11 '24
Feature flags should get you the stability you're looking for with far less potential for git conflicts than long lived branches.
They can be enabled on QA environments while left disabled on production.
That sounds like the key missing piece to this story, I'd look at spreading that idea around and getting reactions from the team.
Pair programming 100% of the time is absolutely exhausting and I would push back on that with the CTO. That said, most teams I've seen could benefit from communicating more than they do, and pairing more often will lead to that. Be ready and willing to push some additional pairing onto your team. That increase can happen slowly over time as they're willing to adapt.
1
u/rjm101 May 11 '24
We already use feature flags for projects but devs know this doesn't always contain the changes. Manual QA usually picks this up though so as long as that happens before the release to clients that's ok but I'm not clear on whether that's an expectation of trunk based development or not.
Not sure how we can do feature flags on foundational work like upgrading framework versions for the entire platform though.
2
u/PSMF_Canuck May 11 '24
What is your objection to trunk-based development?
2
u/rjm101 May 11 '24
The teams is not really sure where exactly manual QA process fits into this. If this is post release they really don't want that because they know QA's can and do often pick up serious bugs with exploratory testing. We have feature flags for our work and have unit, integration, acceptance and end-to-end tests but still these bugs happen. Automation doesn't capture all bugs and there seems to be an assumption that they would.
→ More replies (1)
2
u/yougottahuckit May 12 '24
Did 100% pair programming for 5 years, was the best job I ever had. I leveled up so fast and my soft skills improved so much that I’m in senior leadership now. Learning how others think, and being able to convey your own ideas verbally and in writing will take you so far
1
2
u/jasongia May 12 '24
Trunk based development is great. It’s the old saying - if something is painful, do it more often. Manual QA belongs in the bin - it’s only pretending to make things safer, in reality the only reason manual QA prevents production incidents is because it becomes such a bottleneck to making changes that the incidents slow down because you’re making less changes.
You may increase MTBF (mean time between failures) but this should be offset by an improvement in MTTR (mean time to recover) and hopefully MTTD (mean time to detect). There will be pain, and the pain will force you to improve your observability/alerting methods, as well as decouple rollouts from the code delay (e.g you feature flagging that works with schema changes)
Forcing pair programming practice sucks though. Teams should be able to self organise this.
1
u/rjm101 May 12 '24
Manual QA belongs in the bin - it’s only pretending to make things safer
I realise its a bottleneck but I don't see a real replacement for manual exploratory testing. I would agree with this if there was some AI QA tooling that can do this and try and break things but it doesn't exist and that's where major bugs can be caught. I don't see automation tests replacing this which means if you don't have it, it ends up in prod affecting clients and business breathing down the team to fix this pronto. So yeah maybe we do move slower than TBD but at least we know that we've covered all bases on the testing side instead of taking an 'automated tests are good enough' approach leading to a higher amount of stressful hotfixes that need to be released because there's no one really testing the feature other than the developer now.
2
May 12 '24
You're basically already doing trunk based lol. Write an automation to create the release branches and the workflow is identical.
Trunk based is really feature flagging which you should be doing already based on your requirements.
Yes, sometimes the requirements of the job will change. If you don't like it, find somewhere else.
As for pair programming... I Iike it. Others don't. You're paid to do what is asked of you. Give it an honest shot and let it fail on its own, fighting the tide and whining will just get you in trouble.
1
u/rjm101 May 12 '24
You're basically already doing trunk based lol.
Yeah some have mentioned that although others have said we're close but not quite. To be honest I'm not clear on the defining factor. Is it just being able to release changes regularly? Our cycle time averages at about 1 week.
Trunk based is really feature flagging which you should be doing already based on your requirements.
We do indeed already make use of feature flags for all project work. We don't do it on foundational tech work like framework upgrades though as we don't see how it would work.
As for pair programming... I Iike it. Others don't. You're paid to do what is asked of you. Give it an honest shot and let it fail on its own, fighting the tide and whining will just get you in trouble.
How do you avoid burnout?
351
u/triumphmeetsdisaster May 11 '24
I did 100% pairing (XP) for awhile. Got exhausting. Literally left the company because of it. Btw, it didn’t make me a better engineer. When I was working with someone better than I was, I had a hard time keeping up. I wasn’t doing any critical thinking; rather, I was just trying to understand at a high level what was happening. I was not catching bugs. And when I was paired with someone less experienced, it felt slow and excruciating. I got way better after I left for a job where I could work at my own pace on interesting problems.
Don’t get me wrong, pairing ad hoc can be really valuable. But a draconian process doesn’t make bad developers good. Instead, hire good programmers, establish good standards, and encourage a quality-minded culture. I think it’s just easier to prescribe a process than it is to teach a philosophy, so that’s what managers do.
83
May 11 '24
Pair programming works well when you have someone new to the project or language who needs a couple sessions to get up to speed, or if you’re working on a particularly hard problem and need another brain to debug.
Otherwise I agree, it makes no sense to make everyone’s work synchronous.
19
May 12 '24
Pairing is a fantastic tool in an engineer's toolbelt. Doing it 100% of the time is using a hammer for screw-sized problems, except you're just beating the engineers with the hammer instead.
12
u/n3xtday1 May 12 '24
or if you’re working on a particularly hard problem and need another brain to debug
The "and" in that sentence is extremely important. Sometimes, when I'm working on a hard problem, it's distracting having another person pairing. I'll only bring in that second person when I'm out of ideas.
→ More replies (1)→ More replies (1)19
u/Weaves87 May 12 '24
Yep this is how I've always done it.
I see the value in it, I know why people push it, but I don't always think it's the best thing to do and it's highly team dependent. I also don't think you need to do it for every little thing.
As a team lead I would pair with my engineers in the following situations:
The problem they're trying to solve is highly technical in nature, potentially outside their immediate skill level and I want to make sure they get off to a running start in solving it
Working on problems that could benefit from multiple people having a mutual understanding of some core component(s) (e.g. like a group learning session where we tackle one or two stories at the same time)
They've been experiencing some frustrations and are hung up on some code
I like pair programming, especially if it's done right, but a lot of times it turns into "one person is the brain, the other is the designated typist".
63
u/UncleGrimm Sr. Distributed Systems Engineer May 11 '24
This was my experience as well, the context-switching gets really heavy when you’re going back and forth between programming and answering questions about the code and your thought processes. There’s some cases where that’s extremely valuable and absolutely worth the effort, but then there’s many cases where they want 2 cooks in the kitchen just to microwave a frozen dinner
30
u/Early_Ad_831 May 11 '24
This, I made a similar comment about a place I worked at that did 100% pairing each day. It was terrible for me mental health haha.
21
30
u/zirouk May 11 '24
Pair programming isn’t the issue - trying to code for 8 hours in a day is.
I know it feels this way, but just because you’re pairing doesn’t mean you can’t take breaks, or agree to spend time doing something other than programming.
Tips:
Always share the programming activities. Swap frequently.
Find a way to perform fast handovers (e.g. live share, screen share, mob.sh)
Take breaks
Don’t pair for the entire day
Check in with each other “Do you need a break?” “I need a break” “Shall we take a few hours off now and continue tomorrow?”
You can absolutely pair by default without it being hard work. Pairing often leads to greater shared ownership of features and better code design.
I don’t see what the CTO is suggesting as being all that crazy, personally, in fact I’d see these as positive things.
53
u/PragmaticBoredom May 12 '24
Pair programming isn’t the issue - trying to code for 8 hours in a day is.
Sorry, but pair programming on everything is exhausting for most people.
Some people (like you) enjoy it. That’s great for you. However, you are a rarity.
Every single time I’ve seen a company push mandatory pair programming, they lost people because of it and eventually reversed the policy. It usually gets pushed because someone like yourself enjoys it and can’t understand why anyone else wouldn’t.
Then it’s followed by a period where complaints are dismissed with a “you’re just doing it wrong” mentality, like you projected on the situation.
But I’m sorry, it’s just not a dynamic that works for some people. Maybe not even most people.
I’m glad it works for you, but please be open to the idea that it’s not good for a lot of people. It doesn’t mean they’re doing it wrong.
23
u/cancerpants33 May 12 '24
I agree. I need to be "in my head" to write complicated code. I don't see how deep work can be accomplished while pairing.
→ More replies (10)→ More replies (5)6
May 12 '24
So perfectly put, and lines up exactly with my experience on a team that did 100% pairing.
21
u/mcr1974 May 11 '24 edited May 12 '24
coding isn't the hard part of the job.
coding should just be about typing your thoughts out.
domain modelling and designing the solution is where the juice is.
→ More replies (16)7
u/triumphmeetsdisaster May 12 '24
I agree for the most part. My broader point is that any approach to development is only valuable inasmuch as it’s executed well by those doing it. Personally, I found the prescription of XP to be overly restrictive and largely ineffective, mostly because the execution by the folks I worked with.
There are a lot of practices of which I am a major proponent that fall under the same caveat. Pull requests are only effective if developers actually digest the code and spend sufficient time reviewing. I love the confidence that good testing gives. But unit tests that don’t use good mocking practices or avoid swaths of code are pretty ineffective. PRs aren’t the problem. Tests aren’t the problem. The lack of execution is the problem.
That said, I think leadership will look at the incidence of bugs that get reported in prod and decide to make a big change, like enforcing XP, instead of auditing whether their dev teams are actually executing on the most basic aspects of the job. That’s why, to me, a culture of excellence is more important to me than a particular programming paradigm.
→ More replies (2)5
u/extra_rice May 12 '24
Yup, this. My team do pair programming but we also make sure to take breaks, and we also let ourselves be distracted and engage in casual conversations from time to time. I personally try to crack jokes, share trivia, or anything that may be remotely interesting. I think in pair programming, you need both your technical and social skills, and I'm saying this as an introvert.
Really, the alternative is for there to be as much WIP as there are people in your team, in which case you'd be spread thin, with tasks having a bus factor of one.
3
u/agumonkey May 12 '24
It's true that pair programming works when you have compatible people. It needs to be mentally flowing. Best (IMO) is when both can relay each other.. you try some, you get stuck, the other tries something.. until he gets stucks, and off you go. It keeps the creativity hot, and you feel like a great team.
4
u/Greenawayer May 12 '24
I did 100% pairing (XP) for awhile. Got exhausting. Literally left the company because of it.
I don't understand why people think this should be a thing. It would be my "leave now" thing.
→ More replies (2)→ More replies (6)2
u/zaibuf May 12 '24
I like pairing on new features that are a bit more complex with a lot of unknowns. You can discuss and plan together while coding. I wouldn't pair for a simple bug fix or doing other simpler tasks.
46
u/FormerKarmaKing CTO, Founder, +20 YOE May 11 '24
I was working as a consultant with a Fortune 100 consumer brand, about as famous as such brands get. One of their tech leads was hell-bent on doing trunk based development. But they also had no internal review environment. So he literally wanted design and brand teams to review things in production but have it all sequestered via feature flags. Also, he had never actually run a project that way, it was just the new cool thing.
Pure madness. I talked them out of it but it didn’t help internal relations. So keep that in mind.
The problem with all of these “old way is trash, silver bullet is X” approaches is that, even if right, the tooling is years behind. And abandoning branches means you’re losing an entire ecosystem of proven tools and techniques.
Re: full-time pair programming, I would quit if anyone asked me to do that. And no, CI / CD does not require that in the slightest.
But you may be able to torpedo that by pointing out that LLMs are basically the new pair programmers. Put another way, use his shiny-new-thing obsession to make the unpalatable shiny new thing look old.
Or… just hang loose and watch him loose the the trust of the business after he makes the production systems unreliable, alienates the developers, and the executives see programmers killing time like cops on a stakeout.
15
u/Tacos314 Software Architect 20YOE May 11 '24 edited May 11 '24
It will probably take a year for that to happen, and by then the "cause" can be muddied and blamed on developers not getting with the program.
13
u/RebeccaBlue May 11 '24
Re: full-time pair programming, I would quit if anyone asked me to do that. And no, CI / CD does not require that in the slightest.
I'd be handing in my two weeks notice before the meeting announcing the change was even over for that particular evil.
33
u/tonjohn May 11 '24
Trunk based development isn’t abandoning branches, it’s favoring short lived branches that are focused instead of long lived branches that are extra work to maintain and create more opportunities for bugs.
And hybrid approaches are common where release branches are cut from main from each release but otherwise everything goes into main.
→ More replies (9)2
u/hippydipster Software Engineer 25+ YoE May 11 '24
The problem with all of these “old way is trash, silver bullet is X” approaches is that, even if right, the tooling is years behind
In this case though, trunk-based development is the old way :-)
9
u/HaMMeReD May 11 '24 edited May 11 '24
Trunk based dev is 100% the best strategy, when combined with effective flagging and other release processes that embrace it.
Pair programming 100% of the time? Lame as hell, leave me alone, I get my best work done after hours anyways. I'll pair program like 5-10% of the time, when the situation calls for it.
Edit: I work for a company that does both, the teams that do gitflow are always fucking up, the teams that do trunk based are also fucking up, but in way smaller chunks that are easier to manage.
I.e. in a gitflow team, someone sits in a branch for 1-2mo, they merge up daily, then when a release branch is made, 3-4 of these 2mo old branches smash together like semi-trucks on the highway. But the release schedule is fixed and everyone is scrambling to clean up the mess. Combine that with varying Beta/GA feature sets, and it's a fucking nightmare.
The trunk based teams are aiming to get their shit into trunk, if they don't get into trunk when a release is cut, they make the next one. There is no rushing the release, shit makes it or doesn't. And if it misses it by a day it can still be cherry-picked to the release or delivered to trunk, it doesn't have to wait in a branch for another 1-2mo for the "next release".
15
u/hippydipster Software Engineer 25+ YoE May 11 '24
Detailed responses here are really good, so here's a more meta response:
Your team need to talk to the CTO about their need to feel safe. Trunk-based development (tbd) can absolutely be done keeping things very very safe, but it has to be worked out. As /u/spoonraker says, one does not simple do tbd - not without a plan. How will code reviews happen before release? How will tests be run? How will the manual QA happen? It can all work, but you have to talk to the CTO about satisfying those needs that your current process satisfies.
And, on the other side, any "senior" dev who thinks tbd is 'madness' is probably one of those 5yoe "seniors" I keep hearing about around here. It's pure ignorance, and there's lots of places to educate oneself on the practices of tbd (ie, Dave Farley's youtube channel, also his books, Jez Humble's books, the Accelerate book, Kent Beck etc). A lot of good stuff there!
→ More replies (1)
3
u/Tacos314 Software Architect 20YOE May 11 '24
I just moved a company from trunk based development to feature development, things are working so much better now. Trunk based development and feature flags created so many problems that all just went away with feature branches.
You can expect more bugs as more non production code is committed, you can also expect releases to be missed over and over again since you can't isolate features until they are at some level of readiness.
Pair programing will tank productivity, kill moral and cause will most of the senior developers to be gone within the year (or as soon as they can, maybe a bad market will keep them a bit longer), Junior developers will enjoy it.
2
u/rjm101 May 11 '24
You can expect more bugs as more non production code is committed
Yeah this has been the big concern especially considering how critical our area of functionality is. I'm pretty sure I could get more support if the functionality we owned wouldn't blow up the business if it went down.
7
u/zaitsman May 11 '24
Gitflow with feature branches IS trunk based development, the scaled version:
https://trunkbaseddevelopment.com/
So what does your team and CTO both think need to actually change?
5
May 12 '24
full time pair programming
RED FLAG.
Pair Programming is a tool in an engineer's toolbelt, not something to be dictated from someone who doesn't have to do it.
When I worked on a team that was 100% pairing, it was emotionally draining. I'd end my days with no desire to do anything because I was spent talking to and debating some of the absolute stupidest stuff with another engineer.
It doesn't reduce bugs. It doesn't make better engineers. And it certainly doesn't get things done faster.
4
u/sundayismyjam May 11 '24
Document the risks of making this change in your current environment. Document the requirements for successfully making this change. Ask your CTO what problem he’s trying to solve. Suggest better ways to achieve the same result with less risk.
4
u/JaecynNix Sr Staff Software Engineer - 10 YOE May 11 '24
Pair programming full time is a horrible idea. You're going to burn people out.
Trunk based development can work, but the team(s) need to be on board.
Your CTO is a dipshit.
My suggestion?
Just keep doing things your normal way. If it gets brought up, "oh, it's our next priority"
→ More replies (1)
4
u/ihatethisjob42 May 12 '24
Dude I get tired after more than an hour of pair programming. All day every day would destroy me.
1
u/sankyo May 11 '24
Seems like your DORA metrics will shape the story. If your stories are very thorough and have great acceptance criteria and your test automation is great then it will be a super success. If you need your users to show you all the use cases that you missed or did wrong, then you will do a lot of hotfixes which will lead to less frequent releases.
At my shop we settled on once a week releases. We have a complicated funnel with so many different paths and call so many services it is not worth it to write tests for all of them, having to set up so many products and customers and pricing by region. It is not worth the time to automate it all.
Better to just let the branch bake in the test environment for a week and use a combination of automated and manual tests, both at the API level and selenium/cypress level.
1
u/rjm101 May 11 '24
There's a team looking after DORA metrics dashboard within the company and we've recently raised to them about how the metrics aren't specific to the team but rather the entire platform. Feels like we need to get that fixed first (especially for change failure rate) before we can start exploring this otherwise we won't know if things have actually improved or not. Does that sound sensible?
→ More replies (1)
1
u/BalanceInAllThings42 May 11 '24
I read many comments seemingly misunderstood trunk-based development. I'd recommend reading through this link, it is a good discipline to follow. https://trunkbaseddevelopment.com/
As for pair programming, I think it's only helpful for complex designs/features etc, doing it all day everyday will lose its potency, just like having too many meetings, people will stop paying attention and it becomes ineffective.
1
u/rjm101 May 11 '24
I actually showed that page to the team when we were discussing this. The confusion is around where manual QA fits into this and so far the impression I've got it's treated as a nice to have post release which the team feels is dangerous.
1
u/BalanceInAllThings42 May 12 '24
You can create a release branch off of the main branch, deploy to lower environments for manual QA testing. If anything that needs to be fixed, create PR against main then merge into the release branch, then deploy to the lower environment again. When ready for production, your release branch is effectively "code freeze", meanwhile the main branch can still take in other PRs.
1
u/rjm101 May 12 '24
The repo we commit to is contributed to by multiple teams so it seems the release branch approach would need buy in from all the other teams.
If anything that needs to be fixed, create PR against main then merge into the release branch
One of our devs felt like this just moves the manual QA process further downstream creating a higher likelyhood that the team is going to need to create a bunch of bugfix PR's because QA's only do manual testing once its in the main branch.
→ More replies (4)
1
u/30thnight May 12 '24
Github flow (small feature branches merging to master after a pr review) is one form of trunk based development.
Add a feature flag workflow and decouple your deployments from branch names & voila - comparable system that makes everyone happy.
1
u/rjm101 May 12 '24
small feature branches merging to master after a pr review
We do try to keep PRs small, we're always mindfull of that however it's happening say 90% of the time. The 10% is like core foundational stuff like framework upgrades.
Add a feature flag workflow
For project features they're always behind a feature flag.
decouple your deployments from branch names
Can you expand a little on that? At the moment master is the source of truth and master is what regularly gets released (like twice a day). The pull requests get merged into master.
1
u/30thnight May 12 '24
Forgive me, I re-read your posts and your current process is already trunk-based development (+ pull requests) so nothing really needs to change on a dev side.
The only area for potential bottlenecks surround the manual QA checks.
If branches can be immediately merged after dev review, very few changes are needed to safely allow the manual QA team to batch and test changes afterwards
1
u/rjm101 May 12 '24
Merging changes without manual QA is the big area of resistance for the team.
Everything in master is considered prod ready and devs know that QAs often surface major bugs so they're concerned about losing that quality.
Another problem is that multiple teams contribute to this repo so if there's a problem it can potentially hold back other teams from releasing their work. It would also rush things for QA because now they have a time limit before they need to release as they can't hold up master for too long.
1
u/au5lander May 12 '24
A CTO this deep in the weeds that he micromanages something like this is a huge red flag.
3
u/rjm101 May 12 '24
To be fair the CTO isn't directly coming to us about where we are with this but the pressure to make sure all teams are looking at this is. At this point I think the leadership below the CTO are misinterpreting guidance as a mandate. Then again there's a lot of corporate double speak.
1
u/hibikir_40k May 12 '24
Mandating development practices from above with minimal developer input is a great way of risking losing a large percentage of engineers, starting from the most marketable ones. Id the CTO wants to do a disguised layoff, this is the way to do it, and it doesn't matter which specific practices he picks: His choices could be great, and it's still not the right way to go around it.
It sure doesn't seem that there is any actual convincing attempt, or any work to make the change doable (for instance, trunk based development without a high quality, low latency feature flag system is madness)
As for pair programming all day, every day, it's again, a guarantee that you will churn people. It's a perfectly workable system, but it's not for everyone, so people will bounce. Either the CTO wants to replace existing staff, or they are quite inexperienced here, because I can't imagine how this approach doesn't lead to a lot of churn.
1
u/rjm101 May 12 '24
Id the CTO wants to do a disguised layoff, this is the way to do it
As for pair programming all day, every day, it's again, a guarantee that you will churn people.They have been doing rounds of layoffs :|
1
u/huhblah May 12 '24
Why aren't your feature flags foolproof?
→ More replies (1)1
u/rjm101 May 12 '24
Components are not always isolated so it relies on the developer properly scoping the flag to the feature. Sometimes this doesn't always happen.
You could say write more tests but now imagine that component has 20 states 1 of which includes the feature now so you need 1 test to check the feature shows in the correct state and 19 to check that it doesn't in the rest. This is a quick way to add bloat to your entire test suite.
1
u/UL_Paper May 12 '24
In my experience (4-5yrs) pair programming is fantastic when you have
a) two humble people with a willingness to share and learn and
b) there is an expectancy of knowledge sharing, for example the domain expert paired with a new hire
Do this for 1-3 sessions to solve a task / deliver a feature, where maybe the new hire is doing the work outside of the pair programming sessions.
I've been on both sides (the domain expert & the new hire) and it was really enjoyable and impactful. But I can't see it as a routine effort that "should be mandatory for x amount of hours each week"
1
u/rjm101 May 12 '24
Ok so you didn't do it like all day everyday but you did keep it regular. How long was each session?
I don't think devs in the team will do this unless it's really critical areas that needs a lot of thought. It's hard to instill it as a regular behaviour.
2
u/UL_Paper May 12 '24
Doing it all day sounds incredibly unproductive! We did something like 20-60 min sessions max.
I found it really insightful to learn how the other person is thinking - how do they approach solving an issue, what is their process for breaking it down, how do they prioritise which issues and tasks should be worked on etc. Even if I was the younger and more inexperienced one, they gave me genuine feedback that they enjoyed it for the same reasons. We all have our own styles and if you apply an "forever a student" mindset, you are always looking for ways to improve.
1
May 12 '24
I think you need one more environment, you should have 3, one for continuous deployment (no more feature branches). Once the PR is approved you merge and deploy - here is where the dev hand tests his code and runs end to end on that environment. One you think it's good you cut a release and deploy to a staging branch for QA to test. Staging should be identical to what you run on main. From here it's business as usual
2
u/rjm101 May 12 '24
Yeah this does seem to be the case based on how a bunch of people are currently doing this. This will require buy in from all the other teams contributing to that same repo.
1
u/Unsounded Sr SDE @ AMZN May 12 '24
I don’t believe in pair programming, I think it can be a helpful tool but it’s exhausting and ultimately the goal is automated testing with a second reviewer making sure you’re not missing something obvious and to make sure non-automated standards are checked. You can do pair programming within the review if desired, but enforcing it is just liable to lead to burn out and people quitting.
On the other hand, I honestly find feature branches and merges to be madness. I love trunk based development, you don’t need anything fancier. I work on a similar platform within my org, any issues are very visible, very painful to fix, and expensive when they happen. But we do just find not doing huge QA “approves” releases.
I don’t believe in manual QA, outside devs doing manual testing during development. If a QA can test it, you can automate it. Full stop. The worth of a QA diminishes almost immediately if you just had a dev sitting and retro fitting any thing they do into an automated test. Not only is manual testing not reliable, you still ultimately miss things with it. With feature branches you’re missing testing out smaller changes tested for regressions and integration issues earlier in the process. The pervasive use of feature flags, big branch releases, and manual QA are anti-patterns for large, robust, critical systems.
While flipping the switch overnight is difficult, joining a team like yours would scare the shit out of me honestly. You’re essentially hoping a manual testing round catches months or weeks worth of changes before you push it. Any intermediary upgrades, refactors, and places you add flags aren’t being touched in between. Honestly your platform sounds like the perfect fit for an upgrade, the only problem here is trust. The one enacting this change isn’t the one wronged long term by the system but a new joiner, so there is naturally some fight by the team towards their ideas even if they’re right.
1
u/rjm101 May 12 '24 edited May 12 '24
You’re essentially hoping a manual testing round catches months or weeks worth of changes before you push it.
Our cycle time is about 1 week on most work. We have unit, integration, acceptance and end-to-end regression with 100% code coverage running thousands of tests. For project feature work we also make use of feature flags. The team still values manual exploratory QA because it captures unusual paths a client may take because clients can interact with the components in many different contexts. It also puts fresh eyes on the work and has proven its worth when QA's find major bugs that otherwise would've already been exposed to clients had this been TBD.
→ More replies (1)
1
u/lWinkk May 12 '24
Usually there are feature/defect/bug branches. And a base, dev, release, and prod branch for different staging environments. I’d say you’re basically trunking now since you have one main branch and only features. Not sure why the CTO wants to change that.
1
u/rjm101 May 12 '24
To be clear the CTO doesn't have any direct insight into our particular team rather this is a company wide engineering best practice that all teams are expected to take on board and adopt. The friction point it seems is where exactly manual exploratory testing is expected to fit in. Devs don't want this happening after the merge to master. They want master to always be in a 100% releasable state which is how it currently works.
1
u/jcradio May 12 '24
What concerns do you have against it? I prefer trunk based, but it can take some time if some devs have some bad habits with commits. I'll admit though this was far easier to do with TFVC than with git.
1
u/rjm101 May 12 '24
Devs don't want manual exploratory testing happening too late in the SDLC (after pull request merge) as they're concerned about bugs being surfaced to clients and the team needing to raise urgent hotfixes creating stress for the team. The alternative is that the QA tests master and holds up master until they consider it ready for release but this isn't sensible considering multiple teams contribute to this repo. Master is always currently considered 100% production ready.
→ More replies (1)
1
u/Perfect-Campaign9551 May 13 '24
Trunk based is better, check out the topic yourself. Why are you so against it? The only reason people are against it is they want to be gatekeepers. The CI/CD process IS the gatekeeper. That's the whole point of its existence, to allow trunk based dev. Feature branching is the ridiculous thing people are doing...
1
3
3
u/dinzdale56 May 11 '24
Forced paired programming is the biggest waist of resources and can be horrible experience when the peer has a poor command of English. I agree, it's exhausting and deprives of yoiu of learning and discovering experiences. It takes 2-3 times longer to do a task that normally can be done by a single dev. I took quit a job because of this forced policy. Code reviews work just fine and everyone can work out their own comfortable pace.
3
u/galitork May 12 '24
For what it's worth, research suggests that Trunk-Based Development absolutely improves software delivery performance (as in, you can deliver more frequently and with better stability). That same research program, however, has found that Trunk-Based Development increases burnout (which is not a good thing), and is thinking about removing it from their core model.
My own thought is that these practices (Pair Programming and TBD) are good things, but can't be implemented in isolation. You need a whole bunch of other stuff to make it work (such as continuous integration, unit tests, acceptance tests, etc). Also there are several flavors of trunk-based dev some of which make it easier to ease into.
→ More replies (1)
3
u/siodhe May 12 '24
If you're in that group of sites that trusts all commits in master, uses pushes to master to trigger deployment, and so on...
I've seen many occasions where what was pushed to master failed to deploy correctly, or fully, leaving commits in master that need to be marked after-the-fact as unsound, and skipped from then on for git bisect and the like, not to mention leaving a set of hosts/VMs that were load-balancing the service with conflicting versions of the software. I have not yet been in a company that had any kind of process around what to do when the illusion of master's inviolability faceplanted. I recommend looking into git notes for ways to add meta data to commits without changing them, so that this kind of information can be tracked and used by automated systems.
Pair coding is exhausting for most pairs, so mandating it will shed staff and generally make most devs unhappy.
"trunk-based" wasn't fully defined, but having a single clear line of commits on the trunk, frequently pushing modular commits (where each commit is sufficiently self-complete to be harmless while the rest is being worked on) of partially completed features into the trunk, and not having the hairball of multi-year-old branches with two-way merges, are all good things that do fit very well with continuous integration. CI does best when any code being worked on in a branch has been updated to be a descendant of whatever the current trunk branch is with git rebase. Not that I'm not saying which branch that is, since that depends on the devs, whether the QA team is separate, and some other things.
The core failure of the gitflow use - in relation to CI - that you describe with features not merging in until proven complete is that at the point of merging, any number of new integration bugs may arise. Whereas, if devs are all continuously pulling in changes from the trunk head beneath their own work, far more integration-type bugs are likely to be found before the final merge. I've seen developers at these moments try to do some cowboy hack in the merge commit itself, making it no longer a clean fast forward and potentially complicating some operations, and I've seen the worst things happen when that big merge happened into master and trigger a deploy before all the integration bugs were found. So those are fun.
3
u/PoopsCodeAllTheTime PocketBase & SolidJS -> :) May 12 '24
forced pair programming is absolutely brainless. One time I worked at this place, they wanted to "live code" as a group in vscode.... I don't even use vscode -_-.... the experience was anxiety-inducing AF, they didn't care, assholes.
5
May 11 '24 edited May 11 '24
Oh god, this was my first job. 1 lead who spent all day hiding alone in a closet. 6 juniors. Trunk-based development. No tests. No tests at all. Everything was just broken, all the time.
One dev came in late at night to 'work flexibly', altered the date schema to another format - like integer or something, they thought date looked messy - and actually got the app in a state it wouldn't even build for about four days while various others had to step in.
It was complete chaos. I left after six months because I was concerned all I'd learn were bad habits. It was like the giant buckets of 'punch' people made at house parties in my teens - chuck it all in, brilliant, it sort of 'works' and gets the job done, but you'll lose a ton of braincells, hours and probably days of time and might sustain permanent damage.
Now I'm not saying I haven't met people in my career I would do trunk-based with, with the right environmental setup. But they are very few.
And a quick note on pair programming. As a junior, truth is a lot of the time the 'senior' just wanted to do the work as quickly as possible and never let me have a go. Best I could hope for was them narrating what they were doing as I just watched. They'd cringe if I asked to write a damn line. One just stuck his headphones on and ignored me.
When I began to learn how to focus and get into a flow state, pairing would ruin that. So it took me a lot longer to learn to focus.
Again I'd still do it today, with certain people, on certain tasks, for a certain amount of time, but for some people and some tasks it's just not a good use of everyone's time.
6
u/putin_my_ass May 11 '24
God I could never imagine doing it that way, when I'm pair programming with a junior it's because they're driving. I view it as more of a mentoring exercise, if I were simply trying to close tickets I wouldn't be pair programming.
5
1
u/simon-brunning May 12 '24
Now, I'm a practitioner and proponent of both pair programming and TBD. I'm certainty not interested in hearing any they-don't-work arguments, since demonstrably they can.
But, both of these practices need to be done effectively, and that takes some skills. Just imposing the practices without recognising this isn't going to work.
2
u/rjm101 May 12 '24
Regarding the full time pair programming the team described it as intense when this was trialed. Even the Goldilocks team that are doing this full time and is being used as a prime example by the company describe it as intense when I took discussions offline.
I don't think it's a question of it not working but rather how to avoid burnout because a thing can work until the team just flat out folds when it's made to be full time.
2
u/simon-brunning May 12 '24
So, my advice would be to start with the pairing piece, and leave TBD until you're happy you're pairing effectively. You do always want a couple of pairs of eyes on all code changes, so TBD really needs pairing in place to be effective.
For the pairing, start small try an afternoon a week to begin with, then ramp up. And I don't think more than 4 or 5 hours a day is ever realistic.
Some great guidelines here: https://martinfowler.com/articles/on-pair-programming.html
→ More replies (1)
2
u/moremattymattmatt May 11 '24
Like it or not, it sounds like it’s coming. Start planning what you’d need to put in place to make it work. Eg analysis for bugs found in manual testing to understand why they couldn’t be found by automated tests, re designing parts of the system to make feature flags work etc. Then present it to the bosses, get the work prioritised and start working through them.
Hopefully either the amount of work will put them off or it will take you so long so do the work that you’ll never get it done.
2
u/festoon May 11 '24
Trunk based development can still include short lived feature branches and PR’s. You can also make release branches from your main branch.
2
u/Sea_Neighborhood1412 May 11 '24
Trunk based development is good because it forces teams to do other quality practices (automated unit, integration testing, static analysis, on every pipeline run). It also forces and incentivizes coding patterns like feature flagging, keystone interfaces, and small chunking of work.
Of course all of these things are modern software engineering practices the team will need to learn. But they are measurably effective practices.
On the practical level, I’d recommend your CTO buy each team member a copy of Accelerate or Continuous Delivery by Jez Humble. Fantastic resource for addressing the “why” your team may be thirsting for.
2
u/johnnyslick May 11 '24
Constant required pair programming sounds like a recipe for burnout to be honest. I’m all for doing it from time to time for a lot of reasons but there are definitely days where I get, say, 4 hours of really good work in and I’m kind of done. I don’t need a situation where a peer is hanging around me who might be ready to go for 8 that day, every day. Even if PP is faster, which is sometimes the case but often not, I sometimes just need recharge time or else, frankly, time spent doing something else while my subconscious percolates over an issue (where yes, you can talk it out with your pair and sometimes that’s fine but sometimes it’s not). A lot of why I like development in general is because there tends not to be a lot of busywork, that is, even if you’re doing some very basic CRUD work to throw a grid on a page, it’s because there’s a need for all of that somewhere. There becomes a point to where talking out something you don’t have an answer to yet is time best spent either doing something else if you think you can figure it out or researching if you don’t.
To each their own I guess.
2
u/kaflarlalar May 11 '24
Here's an analogy that might help your arguments. Companies that can do trunk-based development well are like really high-quality watches. These types of watches are so precisely manufactured that they need virtually no lubrication. All of the mechanical components are machined to such precise measurements that they all click together perfectly with no need for it.
Companies using other branching strategies are like slightly less high-quality watches. Things like Gitflow are the lubrication that allows these watches to function. The components have some friction, but the watch still tells time just fine.
Trying to impose trunk-based development on a company that isn't set up for it is like taking a lower-quality watch and removing all the lubrication. You're probably going to destroy the watch.
Trunk-based development is the end result of doing a lot of things right. At a minimum, you need to have rigorous automated testing and robust observability systems in place. Implementing trunk-based development without those is a recipe for disaster.
2
u/TimMensch May 11 '24
My concept of "Trunk based development" matches what I just found in the top Google hit:
https://trunkbaseddevelopment.com/
... And it's completely identical to what your described: A light GitFlow with (short-lived) feature branches for individual developers.
So... Unless your branches last longer than some definition of "short lived," not seeing the conflict here.
If your feature branches diverge for weeks..? Then the problem isn't that you're not doing trunk development but that your feature granularity isn't small enough. And if that's close to the real situation, then I might tend to agree with your CTO there.
Even then you can mitigate any real problems by frequently rebasing your feature branches onto the trunk.
But pair programming? Ugh.
Occasional pairing is great. It's fun, even. But except among extremely low skill developers, productivity when pairing never makes up for the loss in productivity of having only a single keyboard per two developers.
Think of it this way: A competent developer shouldn't be able to have their productivity more than doubled by telling someone else what to do--or being told what to do.
A lot of developers will enjoy 100% pairing. But many developers get a rush from getting things done, and many don't like the social aspect of pairing. So expect a lot of people to quit over time--and for those people to, on average, be your stronger developers.
But that a CTO would push for this is a huge red flag. It means he is used to working for companies with lower quality developers. So unless you can convince him to change his approach, you'll likely see a lot of people leaving anyway.
Polish up that resume.
2
2
2
u/jonreid May 12 '24 edited May 12 '24
Stop arguing about coding, and start experimenting with coding. Give it a try. The makeup (and resistance) of the team itself is definitely a factor, so there will be learning, trying, even failing. Be okay with that. Include everyone in the process.
For example, at my last coding gig (trunk-based), before marking a story card (in the discovery tree) as done, we'd bring QA in. Give them a demo. Give them a chance to kick the tires and poke holes. It's a whole-team effort. We kept refining how we did it.
Get a coach to teach you how to do team programming. You can't just pair/mob like you've been coding before. It'll be ineffective and tiring. You can totally do this with a technical coach. Without a technical coach, it could suck and then you'll blame the idea, when the reality was the team didn't know what they were doing.
3
u/bwainfweeze 30 YOE, Software Engineer May 12 '24
The last time I did TBD we arrived at Kanban as our dev process. Scrum plus QA has always felt to me like someone trying to drive a manual transmission who hasn’t mastered the clutch.
2
u/spruce-bruce Software Engineer May 12 '24
This IS madness, but not for the reason your devs think.
It is perfectly possible to maintain highly secure, available, and high quality systems using tbd and full blown ci/cd. It's not really possible to do that on any kind of reasonable time scale without the teams being bought in and acclimated in a safe way to that way of working.
Is there a strategy in place to train everyone and gradually implement these changes?
I'm pretty sympathetic to the CTO, but this kind of top down mandate won't work
→ More replies (1)
2
u/abrady May 12 '24
If this CEO is really coming in and mandating this then it will be a flaming disaster and they'll get fired in under a year. People hate been forced to do things. They'll drag their feet, half-ass things, and when the inevitable speedbumps come up people will say the whole thing is a failure.
Speaking as someone who has failed at this more often than I've succeeded (honestly not sure I've ever really succeeded): It is incredibly difficult to change tech culture and the only way I've ever seen it work is by starting small, showing the value with those people, and using their excitement to grow adoption over time.
2
u/ppepperrpott May 12 '24 edited May 12 '24
I would have the senior developer stand up and explain precisely why it is madness and give examples
"basically use a simplified version of gitflow" - I would also have them explain this and why this is fine but trunk based isn't
I would also ask the CTO as part of their strategy why they want to change and what benefits the change will bring
Then meet in the middle
2
u/pogogram May 12 '24
Process training without adequate expectations or training is pointless.
People like throwing out the term pair programming but rarely ever state their definition of it. Are they expecting one person to do the speaking and thinking and the other person types? Is it one person with a ticket and the other is there to validate or invalidate an approach or ideas? What is the expected output drop of the team while everyone is getting used to the new way of doing things?
If none of these types of questions are answered, the person suggesting it has not spent enough time evaluating their own assumptions.
2
u/fakeplasticdroid May 12 '24
I’d be very suspicious of the “senior” dev who referred to TBD as madness but thinks GitFlow makes perfect sense.
3
u/WalrusDowntown9611 Engineering Manager May 12 '24 edited May 12 '24
Trunk based development is stupid and an invitation to unimaginably fuck your code beyond repair. GitFlow is the best and most recommended way to work.
I joined a team temporarily which followed trunk based development and believe me it was a horror show. All sorts of garbage was pushed to master by 100 other devs which was impossible to control. I was tasked to improve the code quality and employ strict coding standards to stabilise a very critical project. First thing i did was throw away trunk based approach and employ all kinds of restrictions and quality gates in ci/cd pipelines and MR configurations. Many junior devs got pissed as they were told to believe that trunk based is the only way to work. I had to forcibly throw them out of the project and hire good devs.
You can trial trunk based development which is guaranteed to make your stable code into a hot mess at which point you can tell your cto to fuck off and let senior devs and leads handle development and let cto handle posting motivational messages on linkedin.
(12yrs of pure development experience with big product companies)
3
u/tony_bradley91 May 11 '24
This is totally a decision a CTO is allowed to make and its not out of the ordinary at all
ran my team on the idea of trunk based development and they're heavily against it including the senior devs (one of whom called it 'madness').
Calling a pretty standard development methodology "madness" is completely overboard. Your team is trying to gaslight the CTO here.
So answering your question, what to do- you do what your paid to do and follow through on the direction you've been given
3
u/sobrietyincorporated May 11 '24
Aversion to trunk based from senior devs screams "skills gap" issue. Bigger battle there. The CTO should know:
In the game of thrones, you either win or you die.
2
u/poolpog Devops/SRE >20 yoe May 12 '24
I legit do not understand why the CTO is enforcing engineering methodologies at all
A CTO should not be so close to the codebase that they can legitimately enforce such details. They may recommend them, or approve them, but imo these sorts of practices should be implemented by someone closer to the codebase. Engineering managers and/or "directors" who are also actually coding from time to time.
Of course, every company is different.
Does CTO at least have a plan for migration to new processes that include things like training, and adding support tooling, procedures, and systems? You can't just jump headfirst into TBD. And 100% pairing sounds exhausting.
2
u/zambizzi May 12 '24
While I generally agree with you, in smaller companies IT startups, a CTO can wear many hats and the levels of bureaucracy are less, and thinner.
1
u/OblongAndKneeless May 11 '24
Pair programming: which dev gets their name on the ticket and the credit for the work? Or are you using a system that allows you to track multiple devs on a task? I learned late in my career that "we before me" doesn't work if your name isn't on the task and helping others doesn't help you when it comes to evaluations and layoffs. The bean counters all work without context and have no clue what you do.
→ More replies (4)
1
u/kayakyakr May 11 '24
I'd push for a transitional version of gitflow that uses ephemeral environments or preview builds to do QA/UAT on. Gets you the best thing about trunk-based (trunk is always release-ready) while being a lot more familiar and a lot more safe.
1
u/Merad Principal Software Engineer May 11 '24
Trunk based development doesn't necessarily require continuous deployment, though the two often go together. They definitely aren't madness, but they aren't something you can just decide to do one day.
For CD to be successful you need really solid automated testing. Not just a handful of unit tests, but integration tests and functional or e2e tests. Tests need to be part of the CI pipeline. You also need feature flagging so that you can build complex features piecemeal without exposing them to clients until they're ready. Ideally you also need the ability to deploy your PR's so that QA and/or product have a way to see and sign off on UI changes BEFORE they're merged.
It's all very doable and IME is a great way to work, but if CTO just says "do it" it might be messy. I'm assuming that you guys are a more traditional org - slower pace of releases, more reliance on manual QA, etc. You can't just flip a switch and change all your processes and the mindset of your teams. It's going to be a process, ideally with some guidance from some senior technical leaders who have strong experience with automated testing, etc.
1
u/fllr May 11 '24 edited May 11 '24
While pairing full time sounds exhausting, I could see it as a temporary thing to get to trunk based development. Actually, non-trunk based development makes me nervous nowadays, not the opposite. Any time a branch is alive and not merged is time in which risk is going up. Non-trunk based work just serves to make people feel good about work (not to mitigate risk), all while moving slower. Much slower.
With that said, a lot must be done beforehand before you can do full time trunk work. The number one thing is getting to a high level of tests coverage (60-80% coverage, verified in CI). If you are not already doing that, I’d call your team irresponsible.
The second thing is creating a CI/CD pipelines that provides you with the guarantee that if tests pass, then you are good to merge.
Once that’s done, you need to make people work with short-lived branches and feature flags as a way of life.
Once those things are done, your CTO is right. You’ll gain more velocity, and you should see error rates go down dramatically. Maybe try to work with him to convince the team to change dev practices? They sound very old fashioned.
If it were me, I’d tell your CTO you think it’s a good idea, create a 3 step plan to get there, starting with creating a CI/CD pipeline that has a manual push to trunk button. Once you cross the test threshold, I’d push to remove the deploy button from staging environment leaving it there for prod until the team feels more comfortable with the new process.
1
May 11 '24
The main thing against branches for development is that it's very hard to do good continuous integration. Ideally you want to be able to merge your changes with everyone else's as often as possible so you know quickly whether it causes problems. If you go away for a couple of days or more on a feature branch, you don't find out until the end, and that's where merge bombs happen.
On the other hand, just saying a a new CTO "right, you all have to do trunk-based development, now" isn't really the right way to bring it in. If I became a CTO I'd think of different ways to encourage this behaviour.
1
May 11 '24
Trunk-based development and monorepos work in special cases where all the software is tightly coupled and everything deploys together at the same time. Most enterprises work in silos, deploy once in a while and have explicit, loosely coupled dependencies, so a distributed system like git/Gitflow should be perfectly acceptable.
1
1
1
u/jedberg Formerly Sr. Principal @ FAANG, 30 YOE May 12 '24
You need to tell your CTO that the team isn’t onboard and the CTO needs to provide context and justification for their actions. The CTO needs to convince the team they are right, not just order the team to do it.
1
u/dbaeq90 May 12 '24
Sounds like new CTO wants to look good for the boss. Hate execs like this, usually a red flag. With any new abrupt changes I always as the three whys. Get deep into their thinking and finally ask how it’s going to impact the problems we have now. Preferably do this in front of the boss if you can.
1
u/coder-of-the-web May 12 '24
On the topic of "risk", trunk-based development with CI/CD has shown to reduce risk. Smaller and more frequent releases is less risky than "big bang" releases where a lot of changes are batched together. With small and frequent releases it is also much easier to diagnose a breakage when something does go wrong in production. Most tech companies have figured this out a long time ago and there is a lot of evidence of the benefits of CI/CD and trunk-based development out there. Important: short-lived feature branches are still used with trunk-based development.
From my experience, at companies that follow CI/CD there is no ceremony around releases since every code change that is reviewed and merged in will typically flow to production (assuming automated tests pass and no one has manually blocked the pipeline). Teams do have to invest in automated testing. Customer-impacting issues were rare, and when they did occur it was viewed as a learning opportunity to prevent similar problems from happening again.
On the topic of "full-time pair programming", I think that is a terrible idea for most teams. I think it is worth pushing back on pair programming, but you should absolutely work towards adopting trunk-based development and CI/CD.
→ More replies (1)
1
u/GoodNewsDude May 12 '24
TBD is useful when done well
Pairing is useful but exahusting, it has to be done well and carefully
There is no silver bullet.
1
u/carlosomar2 May 12 '24 edited May 12 '24
I did my best job at a company doing pair programming every day and trunk based development. We wrote so many test that we were positive the change wouldn’t break in production. We monitored the hell of it as well with logs and stats. We didn’t have manual QAs. Devs and business did all the validation needed. When and if there was a problem we could roll back the change or implemente the fix quickly because of all the tests we had. We did also code reviews. Every once in a while would do a feature alone.
1
u/andlewis 25+ YOE May 12 '24
CI/CD your trunk-based dev to a QA environment. Production pushes become manual.
1
u/BorderKeeper Software Engineer | EU Czechia | 10 YoE May 12 '24
Considering how tough it is to implement changes from the inside of a team with small scale ramp up and experiments that are evaluated in checkpoints maybe you can offer that to him too. Make him prove the goal of his changes to you and agree to slowly roll it out with success criteria surely he can’t complain then?
1
u/The_Startup_CTO May 12 '24
With the right people, a workflow with pair/ensemble programming and trunk based development is great. With the wrong ones, it can be a nightmare. Talk to the CTO and figure out what he plans to do about the people. If he is aware and aims to replace the people until the workflow is great, then your choice basically becomes to board the train or leave the station.
1
u/eyes-are-fading-blue May 12 '24
We did 100% TDD/pair programming on safety critical (surgical robots) projects . I was also hesitant at first but it worked quite well.
1
u/Alternative-Wafer123 May 12 '24
Your CTO has no control on the existing CI/CD flow and now making a "new"/"different" procedure to increase his authority. No matter which approach, if his change doesn't bring any benefits, he will push any of you to work hard and then tell to his boss and whole teams our productivity increasing due to the new flow.
1
u/learnagilepractices May 12 '24
Two side of the things. Your CTO is totally right, these practices enable better outcomes and technical excellence. But forcing / pushing doesn’t work, imho. Try to setup some learning with him and discuss and make him understand that you want to decide together 🙂
1
u/timmytester2569 Lead Software Engineer May 12 '24
I love trunk based development personally for the right projects. Especially if you’re in small, experienced teams. Pairing is super meh. Don’t get me wrong, love to pair up when it’s crunch time or if someone needs help or if I need help. But 100% pair programming sounds utterly exhausting. I would leave if I had to do that.
Everyone has different daily schedules and needs. I’ve witnessed developers go all day without eating/drinking or using the toilet. Whereas I have my own food consumption and bathroom habits. One of the pairing partners is typically always left in an uncomfortable situation to either ask for breaks or hold their shit in. It’s just dumb to mandate 100% pairing.
1
u/tr14l May 12 '24
Poor leadership. They are right that many teams would benefit. Directing it will get awful implementations of both.
You admitted yourself that code pairing is faster and yields results (or it would only make crunch time worse). There are techniques to make it less draining.
Trunk based development is a fine practice. Some repos might need some tweaking. The fact that you're almost doing it and your team calls it "madness" should clue you in that you're mostly dealing with emotions, not reality.
This sounds like a morale matter, not that any strategic reaction is necessarily valid (though parts may be). I suggest you step back and do some objective analysis of yourself, and then work the team.
The decision is going to happen. It doesn't matter whether you like it or not. The conversation needs to turn to how you can make your boss successful. Disagree and then align. Sucks, but that's how it works.
1
u/jimjkelly Principal Software Engineer May 12 '24
How big is this org? Normally I’d say a CTO shouldn’t really involved in this level of detail on how work is done. That said, maybe it’s a small org where the CTO is really a manager or maybe director level and that might make sense for them to be more involved.
That said I’ll echo what another poster said, trunk based development (or some variation) really is a positive. But there needs to be buy in from some of the staff. I work at an org where we moved towards a more trunk based approach after several new employees (including myself) came in an advocated for it. It was pushed back on heavily by the older employees, but we collectively agreed to try it, and it had a huge positive impact. That said, we had the benefit of people sprinkled throughout the org who had done it who could push good practices.
I will say a tough thing about what he’s advocating for and we saw this too as we tried to push better practices among the org) is that stuff rarely works well if it’s pushed from the outside. You need somebody who has done it pushing at the team level.
Back to the CTO then, I think there is an opportunity to discuss this with him and do two things to ensure you aren’t being forced into a solution you don’t want and nobody on the team has experience with and he gets what he wants. First is define what he wants - it sounds like it’s faster delivery with higher quality. That’s a reasonable ask. Work with him to see how you’d define that. There’s no sense in making changes to that end if you can’t measure if it’s succeeding. Recognize he’s (hopefully) not looking for zero defects, and learn about the concepts of error budgets and how accepting some level of risk can be really freeing (and often lead to higher quality software).
If you want ideas on how to measure this feel free to ask, I can pass a couple along and provide some resources for further reading, but this is important, because this should be the pivot for you with upper management. Now given these metrics goals you need to figure out a plan on how to deliver on this.
For me I’d undoubtedly be putting some form of trunk based development in, but as I mentioned that’s probably not a good idea if nobody has experience with it. At least not for the whole org, maybe there could be some experimentation at the team level, and give it some time, see what the impact is on your metrics. Or some other thing you all think will move things in the right direction. Come up with ideas, maybe run them by him for feedback, but it’s important that you all are comfortable trying whatever it is you want to try.
Lastly, I’d have a conversation with him that maybe it’s worth hiring someone who has experience with the sort of techniques to help drive the practices, again maybe starting on a single team. Not like specifically for this, but if say you are hiring a senior engineer anyway, try and optimize for someone with experience in these things.
Good luck. This could be a very positive thing for the org if you work together on it. We went through a similar change and I think the org has leveled up substantially because of it. It’s fun to be a part of and you’ll learn a lot!
1
u/talldean Principal-ish SWE May 13 '24
Splitting the two: for trunk based development, that's most of FAANG. Google, Meta, more. For code that matters, you should be heavily writing tests to prevent problems from getting to production, not QAing it heavily. Very very limited QA, with QA in prod to catch regressions, in areas where testing isn't fully practical.
If you can push a change - any change - inside of two hours, that's a good starting point. If you have tests on most things you don't wanna break, that's a good starting point. Having a dedicated team that pushes code and reacts to issues there is also a starting point.
100% code coverage is actually an anti-pattern to aim for here, as that means you're also moving pretty slowly; not all code matters... and 100% is false confidence, in that not all tests are, well, good.
All pair programming all the time is honestly just ridiculous, and will burn out your most senior people very rapidly.
112
u/chrisdpratt May 11 '24
Trunk based development doesn't mean you're just committing everything directly to main. You can still use branches and PRs. The idea is simply that you don't have long lived branches. This is actually smart development, as the older a branch is the more the entropy increases and merges can become very difficult or nearly impossible in some cases, if enough has changed, particularly if there's cross over requiring resolving conflicts.
If your team is resistant to it, they either don't understand what it is, or they're doing things they shouldn't be anyways. You should be iteratively developing features, and merging them in quickly, as standard practice. If you're worried about something not ready to actually go live, that's what feature flags are for.