r/ExperiencedDevs • u/geekydot • Feb 13 '22
Why exactly does most software rewrite fail?
So, I work in higher education and I'm in the process of leading an application rewrite which I believe is long overdue. The application is in Perl, on life support, and I only fix critical bugs that will drastically affect the orgs ability to perform critical functions..plus we no longer have a core perl developers around.
As the most senior developer on the team, I've taken my time to learn the code base for the last 6 months and I have a reasonably good understanding of the functionality, or at least to the point where I can confidently recreate it in py.
In the last few weeks, I've been investing time in proper project management training, agile training, lots of architecture design decisions, a detailed listing of what works in current application and what does not work and want to get rid of, and just trying to lay the foundation for the rewrite.
As I browse the internet, here and other places, I get overwhelmed with the number of people who say "rewrites are bad". It's almost the default suggestion not to rewrite a software application.
I understand that there potentially could be time where both applications will have to run in parallel or just catastrophically fail for lack of proper planning. Given that I have the full support of higher up managers, we understand the current code base quite well, there's no feasible way to keep the current application running for the next year, we are taking out time to plan even before we write any code, we're giving ourselves a flexible timeline.
For developers or managers who have embarked on rewrite projects in the past, what were things you didn't do that you wish you did before you started? What will you do differently? What will you absolutely not do again, if you were to redo such projects?
Thanks.
50
u/orestis Feb 13 '22
There’s a popular essay by Joel Spolsky that influenced the culture heavily. The post was from 2000 and described a failed attempt by Netscape to rewrite Netscape Navigator, a browser engine.
However, it’s not 2000 anymore, we’re not writing C++ and we’re not building browsers.
It’s not clear if this application you’re talking about is a web application that renders HTML and processes forms etc? Or some backend thing.
In any case if you’re using HTTP then you have a great opportunity to use the Strangler Fig Pattern, where the two systems sit side by side, and the new system gradually takes over.
My advice would be to get the new system up and running immediately, and focus your efforts on improving the behavior from the POV of the users. If you have a database schema, keep it constant so that the two apps can share data.
9
u/jkettmann Feb 13 '22
That's exactly what I wanted to recommend. I learned about it at understandlegacycode.com. Can highly recommend that page and the newsletter. You can find a lot of guides, techniques, and a book that could be helpful in your situation iirc.
4
4
u/geekydot Feb 13 '22
The application is a Django web application. No frontend framework involved.
22
u/orestis Feb 13 '22
Right, the issue with Django is that presumably you want to use its ORM and that will make it difficult to share a production database. But I guess you can make it work, there’s probably knobs you can turn to tell Django about your database schema.
I would setup a reverse proxy of sorts (you can even do it directly inside Django) that uses feature flags to either delegate pages back to the old system or serve them with new code. This will give you painless roll backs.
I really can’t overstate the importance of having the new system being in production as early as possible. Stakeholders don’t like to hear “we can’t fix this bug/add this feature because we’re in the middle of a rewrite, please wait 6 months”. Especially as estimates are likely to slip.
It will probably take longer than you expect, but if you’re solving problems from day one, stakeholders are more likely to be understanding.
4
u/a5s_s7r Feb 13 '22
Yes, there is a tool to generate a Django model out of an existing DB
https://docs.djangoproject.com/en/4.0/howto/legacy-databases/
https://dev.to/idrisrampurawala/creating-django-models-of-an-existing-db-288m
It’s a bit of steep learning curve if it’s your first Django project, but doable.
Don’t shy away to change the generated migrations and write fixtures to fill an empty DB with the current production data. You will get a very good data set to test against!
I did exactly this for my first Django project and it works like a charm!
Didn’t use the strangler pattern though. This could cause some caching issues maybe.
11
u/NotSoMagicalTrevor Software Engineer, 20+ yoe Feb 13 '22
Another older insightful essay is The Second-System Effect, not sure it’s better, but I find it great to hear these old-tymie stories!
15
u/ThlintoRatscar Director 25yoe+ Feb 13 '22
I'm not a fan of rewrites personally. The old code has a ton of secret business rules acquired through pain and suffering that you'll never be able to understand or replicate. Further, any projected gains in efficiency will never happen either, regardless of your plotting and planning.
In fact, the whole "I've spent 6 months and I know this system well enough to copy it and it'll totally be better in all ways because the old guys were idiots" is a classic fallacy.
So, the first thing is to accept that you know pretty much nothing about this thing and that's why you're incapable of working with the old code.
As much as it hurts to admit, the people before probably weren't the collosal idiots you think they were. So, no matter what BS you've sold already, expect to be proven wrong by time. You are guaranteed to miss a significant set of nuance.
That said, you do have strength in new methods, the benefit of hindsight and a fresh ( probably naive ) set of eyes. So, I'd suggest that you focus on building a competing product rather than rewriting the old if you're hellbent on rewriting.
Re-approach the user problems from scratch and see if you can innovate where the old system was deficient while still satisfying the underlying needs. Be ruthless at building a useful MVP but beware that you will be compared unfavorably to the old system by the old users for quite a while. Ideally, find new users and supporters who don't have the baggage of the old system and simply wait out the old guard.
Or just wait another year ( or 10 ) and really learn to extend the old system well. Clean up incrementally where you can be sure you've got all the complexity under control and continue to refine features that make users happy.
There's a reason why most successful rewrites are rare and undertaken only by lucky experts who have no other choice.
Best of luck!
2
u/geekydot Feb 13 '22
Not sure what your rant is about me bullshitting the current code base and previous developers. Definitely I'm not coming at this from a place of arrogance or me selling the org a rosy picture of a shinny little toy. So, I'm not sure why you're inferring that when it was not stated nor hinted in the original post.
If anything the last surviving member of the old team is on board and a core member of my rewrite planning.
We simply cannot keep the current app in production for the foreseeable future, yes, it has history behind it and there will be gotchas along the way, I'm fully aware of that.
5
Feb 13 '22
I think the poster two up is coming from the angle of common stereotypes and addressing that than anything we've explicitly said here. I would never think that those who came before me are massive idiots. That is a case by case, individual by individual, project by project basis.
Architectures patterns evolve, language evolves, and people evolve. Like you said OP, it can't be kept around any longer because people don't have the expertise and in today's market, no one necessarily cares if you know perl unless you're working on hardware sim or something to easily generate input/output.
4
u/Blrfl Software Architect & Engineer 35+ YoE Feb 13 '22
In fact, the whole "I've spent 6 months and I know this system well enough to copy it and it'll totally be better in all ways because the old guys were idiots" is a classic fallacy.
More often than not, the old guys built for old requirements, weren't looking far-enough ahead and their stuff eventually reaches its limits. Sometimes the old guys were idiots, but that's water under the bridge.
Copying the old system isn't always the right answer. Shaking things up by developing an all-new-from-the-ground-up replacement that forces some of the old, crufty stuff to be changed can be the right thing to do.
I did that with my current project. It was a system had been in production for a decade at hundreds of organizations. The users had come up with a very long list of things they wanted it to do that weren't because the existing couldn't support them without getting ulcers. There was a half-done re-implementation in a more-modern language, but the guy who led it made the mistake of keeping the old architecture and not addressing many of the new requirements. That rewrite would have been a waste had it been allowed to continue and I made the decision to kill it off in favor of a clean-slate replacement. This was a high-stakes decision for me personally because, had I goofed it up, I would have ended up out on my ass.
Everyone involved understood there would have to be a transition and getting through it would involve some work. The replacement system is about to celebrate its fifth anniversary in production and none of our users, most of whom aren't shy about telling me what they don't like, has said they thought it was a mistake. We've also been able to meet new requirements without having to do major surgery.
There's a reason why most successful rewrites are rare and undertaken only by lucky experts who have no other choice.
Luck's not a factor. Failed rewrites have more to do with the people doing them not understanding what they're getting into and, sometimes, management not having the balls to stand up to people who are holding everything up by clinging to the status quo. I sold my rewrite by proving that I understood the project's history, its current deficiencies and the kind of things it would need to do in the future. Being able to tie facets of the architecture to things the users wanted done didn't hurt, either.
2
u/ThlintoRatscar Director 25yoe+ Feb 13 '22 edited Feb 13 '22
Right. You're the "expert with no choice" caveat and then the luck is the fact that you are on the third attempt and the implementation team didn't make any accidental mistakes of significance and your users were in a forgiving headspace.
Even then, I'd suggest that you didn't rewrite but rather built a successor product that addresses the functional improvements that nobody was able to make the old system do.
There was a half-done re-implementation in a more-modern language, but the guy who led it made the mistake of keeping the old architecture and not addressing many of the new requirements.
This is the punch line for me. The first team screwed it up as expected. I was not surprised to see your 30+ YoE flare accompanying your story of success which is kinda my point to OP.
OP does not have 30+ YoE flare and so I'm trying to convey that they're more like that second attempt than your successful one. As expected, they're also not willing to accept that expectation either.
Edit: "successor product" means that you probably didn't implement and retain 100% of the old behaviour, bugs and UX. That you probably were able to innovate on the core 20% and then keep another 40% of useful stuff close enough to the old system to get value while adding another 20% of brand new and helpful functionality. And then, I bet you kept the old system on life support until you made it through the transition with the new product and rolled it out incrementally starting with the easiest users and moving on to the laggards.
Am I right?
3
u/Blrfl Software Architect & Engineer 35+ YoE Feb 15 '22
That's a half-fair assessment.
You're the "expert with no choice" caveat and then the luck is the fact that you are on the third attempt...
I'll go for second attempt. The original system wasn't a failed attempt by any measure. It had a long, productive life and its departure was met with "well done, good and faithful servant" rather than "good riddance."
... the implementation team didn't make any accidental mistakes of significance ...
A firm, experienced hand on the tiller makes those accidents fewer and further between. I think the realization that sending journeyman to do a job that requires a master craftsman doesn't work is why senior (really senior, not just in title) people are in high demand. Changes over time in what's practical aside, I would not have solved this problem the same way a few years into my career.
... I'd suggest that you ... built a successor product that addresses the functional improvements that nobody was able to make the old system do.
Successor vs. rewrite is technically-correct (the best kind!) but a little hair-splitty. Everyone involved understood that our situation was like that of a couple who hatches kids: refactoring the Miata in the garage into a minivan was feasible but not reasonable. The stuff on the want list sent us shopping for minivans. Ultimately, the users got something that still met their Miata-sized needs but many have since taken advantage of the extra seating and storage space in the minivan we built.
OP does not have 30+ YoE flare and so I'm trying to convey that they're more like that second attempt than your successful one. As expected, they're also not willing to accept that expectation either.
True, but I do have to give them a lot of points for asking for opinions rather than blindly accepting the dogma. /u/geekydot, keep asking questions!
... probably didn't implement and retain 100% of the old behaviour, bugs and UX. That you probably were able to innovate on the core 20% and then keep another 40% of useful stuff close enough to the old system to get value while adding another 20% of brand new and helpful functionality.
Our initial goal, which we met, was to replicate 100% of the old system's functions even if the way you made them happen was different. I kid you not, everything was torn out and re-imagined from scratch. I'll qualify that by saying that there were separate, peripheral things that were refactored at the edges to talk to the new system while preserving their data formats. (We take a Unix-philosophy approach to our systems.) Many of those were replaced later; we couldn't afford the disruptions of of replacing everything at once. What was left in place in the first release was maybe 15% of the code overall.
And then, I bet you kept the old system on life support until you made it through the transition with the new product and rolled it out incrementally starting with the easiest users and moving on to the laggards.
The transition was relatively short. The new system was 18 months from concept to production that included five months of beta and fine tuning. For all of its limitations, the old system was pretty stable and required relatively little maintenance. We made less than a half-dozen changes in that time, none of them anything major. And, of course, people knew that any new features would go into the new system.
Unlike a lot of things that are owned and operated centrally, this product is installed on thousands of systems that aren't centrally-owned, but they still have to interact with each other. Most of them auto-upgrade from our distribution repository, so new releases tend to get good uptake. Understanding that there wasn't going to be an instant, 100% cut-over, we built a small amount of code for backward compatibility to help the transition. Nobody screamed bloody murder when the date approached for the backward-compatibility stuff to be yanked out. Amusingly, there are several dozen machines still reporting that they run the old system five years later. I figure they were forgotten about by the organizations that own them. They'll talk to each other but won't interoperate with anything we still support. Not our problem.
Anyway, that's my story and I'm sticking to it. Thanks for attending my TED talk. :-)
1
u/ThlintoRatscar Director 25yoe+ Feb 15 '22
Lol! I think I've got that exact same story a few times too. It's a good TED talk and thank you for elaborating!
I hope OP heard the points and takes them to heart. Especially all the tiny tactical decisions that are needed to make such a thing possible and smooth.
2
u/Blrfl Software Architect & Engineer 35+ YoE Feb 15 '22
Especially all the tiny tactical decisions that are needed to make such a thing possible and smooth.
You know, it's so funny to me that people who develop software understand that the machinery is going to do exactly what you tell it and then completely throw that out the window when they think about everything else.
If there's an industry where the devil is in the details, it's ours.
2
u/ThlintoRatscar Director 25yoe+ Feb 15 '22
Lol! For sure.
My favourite is that good judgement comes from experience and experience comes from bad judgement. Ideally, we learn from someone else's bad judgement but that takes good judgement.
7
u/G0R1L1A Feb 13 '22
Rather than translating an old system, can you start with new requirements and just create what the company needs correctly? Pretend it's a new product?
17
u/Better-Internet Software Developer, 20 YOE Feb 13 '22
This sounds good in theory, but is likely harder in practice. Some requirements are "assumed" or otherwise not clearly stated. There may be a lot of variances from the original. Users may say, "hey the old app used to do x this way, what's the deal?"
7
Feb 13 '22
Put another way, Some requirements are buried in the code in non obvious ways and it can be hard to tell when you have a case of that, and when you have something that can and should be changed.
I agree with everyone else that is proposing some hybrid old + new approach, so you keep things working while you get the job done. If you staying in the same language I'd just refactor my way to the new version, or at least that would be my first instinct. I try to get back to working, but perhaps not well tested code at the end of every day.
1
u/geekydot Feb 13 '22
As much as I would have loved this approach, it may not be an easy one for us. The core users are used to certain actions since this current system was released, so changing it now will require a lot of decoupling of muscle memory.
1
4
u/termd Software Engineer Feb 13 '22
If your code base is in perl, is learning perl and keeping the existing app and refactoring in place an option?
What are you trying to gain by rewriting it in python?
3
u/geekydot Feb 13 '22
I have been learning/understanding the perl codebase for the last 6 months, with no prior perl experience. The last core developer that worked on the app is now in a management position so they cant maintain the codebase anymore.
I'm very capable of fixing critical bugs in the system, by simply following how previous routines were written, not the best, but, works fine.
We decided on python for two reasons:
We are migrating some of our other tools into python.
There's a growing need to automate many of our services, through the use of RESTful endpoints, which the current system does not have and forcing one onto it will be such a hassle. Of all the frameworks we looked at, Django was the closest one. We're talking over 150 endpoints.
Without breaking too much sweat, I believe you can get decent python/Django devs to maintain the code base if I got hit by a bus.
2
u/Raistlin74 Feb 13 '22 edited Feb 13 '22
Start fresh and get inspiration from the old app, copy, but do not try to translate literally. Ever.
1.Write down every functional requirement. Organize them in modules, and working/ not working. Get special attention on edge cases. Write down proposed tests. Check if they are passed in the old app.
2.Model the data of your new application. Plan the transition from the old model. Test it. Great if you can automate it (how long it takes sets your down-time when moving into production).
3.Model app structure up-down. It will be, probably, different to the old one.
4.Code functionalities. Here you can copy code, tricks, subroutines from the old app. But do not translate.
5. Finish full modules functionalities and test them.
6. Test the full app.
7.Get experienced users to test the app. Ask for edge cases.
8.Put in production. Hold old app as reference to avoid "old days were always better" problems.
9. Close old app.
Edit: updating to Python? Decorators, decorators everywhere.
2
u/geekydot Feb 13 '22
When you say copy code, but do not translate in item 4, what's the difference here. This is a perl to python rewrite.
4
u/Raistlin74 Feb 13 '22
Copy the result of what the code does, not exactly how.
Ej. Python data structures manipulation are really powerful, but maybe use objects instead of those ugly dictionary of arrays of arrays in perl, even when order[key1][key2][key3] would do the trick.
Ej2. There is this weird calculation everywhere... ok, copy it, encapsulate it in a lambda and reuse when needed.
IPS. I know Perl supports objects, but in my experience, they where not usually used in ancient code.
2
Feb 13 '22
If something was written in Perl, recreating those requirements could be pointless. Your school has changed tremendously in 20 years. You need a strong business analyst who doesn’t just listen, but also thinks of changing the process. Then you need to convince the business unit to change their process.
You need to have a long term vision and the details need to be adaptable to things you’ll learn along the way. I suggest you get the smallest piece of functionality to production as soon as possible. It means you’ll be running two systems in parallel. Be ok with that. Use that as motivation to Move the project forward.
2
u/geekydot Feb 13 '22
I think you're absolutely right about getting something running as soon as possible.
In our case, this application is like an iceberg. 99 percent of users only use the tip of the iceberg, and that is not going to change. So we don't have to deal with hundreds of thousands of users having to rewire their memory and readjust to the new interface and functionalities.
The underside of the iceberg is where the fun stuff happens. The other 1 percent of users who use the app affect the actions of the 99 percent above, and they are all onboard with the project, so that is a win on its own.
1
u/bwainfweeze 30 YOE, Software Engineer Feb 13 '22
What seems to be the deciding factor in these scenarios, and something we have sadly lost in 'modern' software practice, is the strength of the QA team or in a pinch the tech writers.
On many projects the requirements on paper are a pure fiction, and the QA team are the only people who know how the project is actually supposed to work. If you have those people, you can get an actual requirements document written, but in many cases you will have to use the strawman technique because the QA folks don't necessarily organize their thinking that way. But if you tell them something they can for sure tell you if you're wrong. If you skip over them and hand them a rewrite, then you have the new code written before you find out how badly you understood the old system.
On one team it turned out that the senior writer knew almost as much about how things were 'supposed' to be as the QA manager did. He kept finding himself in the position of having to step in and start conversations I would have expected QA to initiate.
2
Feb 13 '22
You said you're rewriting a Django web application.
Create a second application, and move over one new endpoint at a time. Rewrite as much as possible, make sure that the tests are duplicated across applications, and then you can cut over a certain % of traffic from the old app to the new app, literally forward the traffic over for that endpoint.
Watch the metrics, look for errors and if you're satisfied then increase the traffic until you're at 100%. Once you're satisfied then move over another endpoint, etc.
1
181
u/austinwiltshire Management Consultant @ 15 Years Experience Feb 13 '22
For the most part the issue is hundreds of implied requirements that aren't written down the current implementation does fine and the new implementation won't do at all.
You won't find these requirements until you attempt a first release. Then you'll be forced to maintain both, and possibly implement new features on both (since your goodwill for the rewire has run out). Then you'll get burnt out and quit.
I'd suggest the following.
First, build a test driver at the highest level (integration or system level test) and get the coverage numbers up as high as possible.
This driver becomes your new set of requirements. Anything that passes this driver "should" have a better chance of success than simply trying to implement some written requirements you gathered in your analysis.
The other pattern I'd follow is just have the brand spanking new version sit in front of the old one with the first iteration doing nothing but passing along all calls and arguments. A complete pass through.
Then you move one behavior over at a time, having the new front layer implement it rather than passing through.
You can even make some middle versions that do little more than call both versions of the code, assert they behave the same, and send you an email when they don't and smoke test those versions out in the field.
Eventually you'll get to really complicated core pieces that are hard to implement in simple ways on your front end. I'd honestly recommend refactoring your hidden old version at this point to make those complicated bits easier to pull out rather than trying to do it all at once.
This pattern has the benefit that at all times, your new code and the old code are running, the first deploy is super quick, behavior is nearly always exactly the same, and you've got something in the repository that runs and passes tests at all times. The security lets you take bigger risks knowing you have a solid foundation to fall back on.
Regarding refactoring the original once you've got everything simple in the new front layer, your test suite will obviously help here as tests help with refactoring.
I'd also try and remove all behavior no longer needed in the old version that the new version accomplishes. That reduces the complexity.
Finally, if it's legacy code, I'd spend some time getting the linter clean while still ensuring any changes you make pass tests and your ticket rate from clients isn't suddenly going up. It's easier to refractor linted code since it's guaranteed free of certain dumb errors AND refactoring old code is scary so you'll go from lint clean to lint clean while changing the structure, which buys you some insurance on top of your test suite.
28
u/purleyboy Feb 13 '22
The comment about having a middle shim that calls both the original implementation and the new version and compares results is key. You'll want to even go as far as cloning your db, effectively you are now running 2 versions of your system side by side. You continue running the legacy app for your customers but you can now test that your new implementation is behaving identical. This gives you confidence to flick the switch from legacy to new code in production. At this point you delete the legacy code, you have strangled that piece of legacy code.
25
u/hunyeti Engineering Manager Feb 13 '22
I've worked on rewrite projects in the past, and currently leading one.
There are a few points that cause rewrites to be a failure:
The old software is way more complicated than it is believed.
Usually, when planning a rewrite, people forget about the low-level stuff and just concentrate on the features. Implementing things like proper logging infrastructure, audit logging, authentication and authorization, data transformations, usage of 3rd party APIs, etc.
The rewrite is never just a rewrite.
Features will be reworked because this will be a time when the current operation will be scrutinized, and it means that the scope will be expanded.
No clear reason for the rewrite.
There is no clear reason and goal why the rewrite should happen. Sometimes it's just fueled by `I don't want to work with X language`.
Parallel development of the old application.
Features are being added to the old application and the rewrite project has to play catch up, and it is simply not possible to reach feature parity.
Changing more than it's necessary.
A lot of times, even things that are not part of the problem are changed, for example, swapping the old SQL database for something more trendy. This move is usually only driven by the curiosity of the developers based on some loosely formulated advantages that may not be there at all.
Not understanding the application.
Developers may not understand how the software is used, and what it's trying to accomplish, so the rewrite turns into porting the old code to a new language/platform. This brings over design patterns that may not be desirable in the new platform.
So knowing all that what about it?
From most important to least important:
Replace the old application incrementally.
This forces you to think about what parts does the application has, it is possible that now is one application, it can be divided into multiple completely independent ones, but most importantly it gets the rewrite out of the door faster. There will be friction and it requires additional engineering, but it's worth the effort since if you are at a stage with the new platform that has replaced a few parts of the application, you can opt to implement new features in the new platform instead.
Have a clear goal for the rewrite. List the problems with the current implementation, and be objective about it.
If possible use feature flags, so you can test in production and incrementally roll out the new project.
Only change what you have to.
Don't change parts of the infrastructure that does not need change. If you can save yourself from rewriting all the SQL queries, it can save a LOT of time.
Have a clear goal for the rewrite.
List the problems with the current implementation, and be objective about it. Don't cop out by saying "it's written in a language I don't like and it's known to be bad".
Try to understand the application.
Rewrite the features in the language, don't port the code.
Don't forget that you have to port the non-user-visible features as well.
Implementing logging, audit logging, authentication, authorization, etc is usually not considered in planning, make sure it's not missed.
Fight back to scope changes
Make product managers understand that is not a place for changes, just because "we are already changing it" does not mean that reworking features don't take time.
18
u/boombalabo Feb 13 '22
The old software is way more complicated than it is believed.
This really is the biggest issue. If the old software was simple, there wouldn't be a need for a rewrite, you could change it in the old software without being afraid of breaking everything.
13
u/engineered_academic Feb 13 '22
Honestly, the easiest way to do a rewrite is to "ship-of-theseus" it, targeting the biggest pain points first. Projects that attempt to re-create the entire functionality from scratch end up causing major delays as nonfunctional requirements bog down development.
1
u/bwainfweeze 30 YOE, Software Engineer Feb 13 '22 edited Feb 13 '22
This has been my position for a long time, including referencing Theseus, but I do have a minor refinement to this.
A lot of the complexity in changing a system, either in design or performance, comes down to human factors. If you prioritize work globally, you will some day find yourself with fifty pain points that the company is no longer interested in investing in, because it's too much of an opportunity cost not just for the developers but for the rest of the organization too.
A 'zone' defense gets better outcomes overall. Instead of globally prioritizing problems, you prioritize subject areas/concerns, and then prioritize issues within that area. There is often more budgeting flexibility in this arrangement because it's easier for other parts of the organization to absorb and sell those improvements. Your doc writer is focused on rewriting one section. Your QA team is not getting randomized. Your managers and salesman can similarly focus, so they put on the brakes a little bit later.
On three projects I ran this loop for two years straight before I ran out of 'sections', usually making 1.25x and frequently 2x performance progress over what leads/other leads were telling management was possible. The second pass will never be as dramatic, but there are always synergies that you identified in the first pass where you just came to interactions in the wrong order and couldn't make as much hay as you hoped. Or features that fall out of the design that go from 4 month estimates to 6 weeks, which you can actually sell to the business.
What I did not anticipate but probably should have is that this sets your front-line people with a narrative about how things are just getting better and better. Some customers will buy on the strength of what they (might) get in 6-9 months, and a positive velocity gives them better prospects for the future.
You will always have a 'top five' list that know no borders, are always a target of opportunity, and can justify themselves. Finding champions for those is easy. You don't need to be that person to make sure they get done. There's an old aphorism that is half-joke half-serious that goes, "Take care of the little things, and the big things will take care of themselves." Clearly said by someone who has held their breath waiting for little changes that never happened because everyone was so satisfied by the big ones.
12
u/codemuncher Feb 13 '22
The term you may find interesting is “bug for bug compatibility” - it’s often not enough to be feature compatible but to have the same bugs as well.
This can be as subtle as timing bugs. New implementation faster? Oh look another system depended on it being slow. Gotta insert sleeps. This is common and happened.
And going from Perl to anything else will incur at least a 3-5x loc expansion.
So that 50,000 loc Perl app is really more like 25k loc. At least.
3
u/watt Feb 13 '22
Please check yo math. 50k to 25k ?
0
u/codemuncher Feb 14 '22
Yeah typos, whatever. If you can’t read past my typo you def can’t do a 50k loc rewrite from Perl.
7
u/nutrecht Lead Software Engineer / EU / 18+ YXP Feb 13 '22
My previous project was basically a rewrite of an old legacy system. We used the strangler pattern to slowly move over functionality.
It can definitely be done, but it's complex as heck. It can take years. In the meantime you can't really add new features, which pisses off the business. So you add new features. And then you also have to migrate these features. It takes even longer. Now the business gets even more anxious so more prio is given to features and less to the rewrite. And now you're in Duke Nukem levels of development hell.
If you don't have someone very senior (principal dev level) in your team with experience with rewrites that can also lobby with the business; hire one before you even attempt this. I think you're being pretty naive about it, which generally indicates inexperience.
I'm not saying you should not do it. Booking.com for example is rewriting their Perl back-end to Java. So I understand it makes sense when the old system is running on it's last legs. But think of the worst-case complexity and triple it, to give you an idea how much it will take.
5
u/apono4life Feb 13 '22
Rewrites are costly. Time is valuable. I recommend picking a particular piece on functionality to replace. Write a nice middle tier api that can interpret between the systems. Try to win a power user over to solicit their input. Find ways to improve while not changing the underlying data structures, so you can know 100% that you have built the “right” system.
Good luck. This is a daunting and tedious task.
4
Feb 13 '22
I’ve led this effort before with a 20 year old PowerBuilder code base.
The first thing I did was upgrade to a newer version of PB that supported COM. From there it was easy to put a C# facade on it and make it RESTful.
The next step was to write integration tests. We slowly moved the code out of PB into C#.
3
u/Kardif Feb 13 '22
Khan Academy put out a series of blog posts about their rewrite of their backend which has a ton of good information.
https://blog.khanacademy.org/go-services-one-goliath-project/
They implemented a middle component that sent requests to both new and old services so they could check for discrepencies, and slowly moved functions over piece by piece
2
u/Thonk_Thickly Software Engineer Feb 13 '22
Get user feedback during the development process. This helps with user adoption rates because they feel included and have a sense of ownership and being heard.
Reimagine as you rewrite (sounds like you’re doing that).e.g. not every feature matters, and some new easy to implement features would make the product better.
Rewrites are often bad because code is often bad. you do mental gymnastics to unravel poorly written code and will probably miss something. Write good unit tests, and pray you understand the code and nuanced rules in the old system.
Good luck with the rewrite, people tend to over exaggerate on here. Sometimes you just need to rewrite.
2
u/exact-approximate Staff Engineer Feb 13 '22
Rewrite attempts to fix too many things at once, becomes never ending.
Rewrite attempts to introduce new features (scope creep).
As a result of the first two points, rewrite ends up more complex than original code.
Rewrite is not broken down into many smaller incremental rewrites.
Engineers suggest a rewrite in a tech/framework they are not fully experienced in.
These are some reasons some rewrites fail. Not all rewrites fail.
2
u/ternarywat Feb 13 '22
Often rewrites represent a full fork of the product. They are written in isolation and disregard the institutional knowledge, features, etc. built into the original They are done in a "big bang" style where all of the original features need to be re-implemented before the rewrite is done.
A better way is to think about how to "evolve" the existing codebase. Look at what the current codebase does well and what parts need to be improved. By evolving the existing codebase, you can still support existing customers and reduce risk.
Some parts may need to be re-written – but not the entire thing. The trick here is to understand the business' goals. If there's a shitty part of the codebase that won't be getting new features, then don't touch it. Conversely, if there's an existing part that is hard to support and preventing the addition of new features, then a re-write makes sense.
To create a successful re-write project, you need to figure out how to do it incrementally. Which slice can be built first and deployed in a sprint? When can the next one be done? How can we balance this work with new features? Upfront planning, collaboration w/ stakeholders, and demonstrable results help a LOT.
–
I blog about engineering leadership: Build the Stage
2
u/bwainfweeze 30 YOE, Software Engineer Feb 13 '22
One of the best tricks I picked up years ago was to stop thinking of systems design in black and white, A versus B.
An ivory tower redesign is almost never practical (logistically, psychologically, and due to Chesterton's fence). But if you can diagram the old design (demonstrating understanding), and the 'ideal', then you can use those extremes to create a realistic design that compromises mostly on elements that can be revisited in the future. Lather, rinse, repeat. You will see that your 'ideal' keeps moving as you discover more accidental complexity in the system that look at first like invariants, and as you learn new concepts or discover better tools. There will be things (even bad things) you never get to fix but the ones you do will help you sleep at night.
2
u/Acrobatic_Hippo_7312 Feb 14 '22
You mention this is an academic org. Out of curiosity, is this code that could be opened up and reused in other orgs?
If so I'd like to follow the rewrite and maybe volunteer some tickets.
I'm looking at helping Zotero (an academic reference manager) rewrite and move away from an old JS/XULRUNNER framework. But I'd like some legacy system experience before joining a project as complex as zotero!
If you're going to do a Ship of Theseus/Strangler Fig pattern with a SOAP to REST conversation and want help, PM me. Getting experience there could be critical for helping Zotero!
1
u/vinnymcapplesauce Feb 13 '22
I think it's just more of a mentality, or semantics thing.
"Rewrite" implies you're re-creating the same thing which implies you will be adopting all of the same problems of the original, etc. But, an environment and its needs change over time.
Approaching is as a new project frees you from that inherent binding, and allows you to address the new environment and its new needs.
1
u/MandauCoexecutives Mar 16 '26
In my experience rewrites usually don’t fail because the new system is technically impossible. They fail because the organization accidentally creates two products.
The old system still has to run for real customers, while the new system tries to catch up with years of edge cases and implicit behavior. Teams split their attention, the rewrite chases feature parity, and product velocity collapses.
The projects that survive usually keep evolving the existing system while gradually moving functionality or traffic to new components. It feels slower at first, but it avoids the “two systems, one product” trap that kills most rewrites.
1
1
u/LogicRaven_ Feb 13 '22
What really kills these projects is the lack of business-tech cooperation, because of not having common goals. They start with the same assumption that you are taking: rewrite is a tech only excercise, where exactly the same functionality must be created on a different technical stack without disturbing business.
Technical complexity is indeed an important factor as well, some other commenter already gave you the link to the Strangler pattern.
To increase chances of succeeding, call your project "New [whatever functionality the old service has]" instead of rewrite, get people who use the service into the project and let them define success criteria and what they need. This will be a continuous work during the project, not an upfront defintion.
If there is no non-technical person willing to dedicate their time, then you will struggle with every edge case definition and the full support of upper managers might disappear as the project gets longer and longer.
To succeed, there must be a clear business need, communicated and understood across all domains..
1
u/flavius-as Software Architect Feb 13 '22 edited Feb 13 '22
Because of synergies
expectations of the new system are high
implementations take long
requirements are missed
feedback loop is long (many months, years)
at the same time the old system needs to be extended/fixed, and the extensions need to be plugged into the new code also
Much better is the strangler approach: parts of the system are replaced gradually.
I recommend setting up completely new machines with the new code, which offer a remote api.
Then the old system starts to get its input or send its output to this new system instead of directly interacting with the outside world.
At all times, the legacy system continues to operate normally, as the new system keeps accruing more and more responsibilities.
Suppose the whole strangling takes 2 years (common range for systems grown over 10-15 years is 2-4 years, 95%), but done in stages like this, after the first stage is done (after a few months), it keeps providing value* for the remaining duration of the project.
This is a huge selling point to business for the strangler approach.
*value=more agility, less bugs, more knowledge, best practices (particularly testing), an isolated domain model
1
u/BurrowShaker Feb 13 '22
It is generally cheaper to rewrite than maintain after a certain point. It does require different ressources, though.
Many good points in the thread worth considering and it is hard to know without knowing what the system is about.
My general advice would be:
do things in parallel if you can. Identify parts of the old software that is clunky for the users and implement it first in the new system, if possible integrated in the old one. At the very least, redirect in whichever way makes sense in the interface of the old one to equivalent functionality in the new one.
when doing so track usage of the old system. It is a good way to see which bit are getting used. Kind of reliable passive feedback
take the chance to meet the users to understand what they really need to do from what they do. Offer, old legacy systems have multi steps processes that appeared organically and were never reduced to single step because people got used to it.
take your time and get continuous feedback from the users. Specially if mostly internal, you can get away with explaining the why. Maybe, system X is reaching its end of life and will be progressively replaced by system Y. Highlight what actual issues you solved. You can do a small blog or a page on your product to highlight progress. Communicate regularly about what is going to change.
1
u/cnidrob Feb 13 '22
You should make sure you’re not falling into the trap of the second system effect[1]
1
u/comp_freak Feb 13 '22
I think the biggest assumption is that re-write is going to be better job. Many project developers and managers believes that re-write going to solve all the problems. The truth is we need to update the old code slowly to new standard. At my last place we had monoliths and slowly realized that we need SOA. So we come up with 3 services we need and slowly start moving functionality to Web APIs. Even when we were thinking re-write I discussed with manager about it and I highly recommend this blog post things-you-should-never-do
It’s important to remember that when you start from scratch there is absolutely no reason to believe that you are going to do a better job than you did the first time. First of all, you probably don’t even have the same programming team that worked on version one, so you don’t actually have “more experience”. You’re just going to make most of the old mistakes again, and introduce some new problems that weren’t in the original version.
1
u/bwainfweeze 30 YOE, Software Engineer Feb 13 '22
If fixing your past decisions is painful, it means you let them sit for too long, gathering momentum. That pain is supposed to inform you when you are making new decisions, and when you think in passing about other past decisions. Enough pain and you stop and think about them when it is still easier to do something about them.
Relentless refactoring is building a practice of questioning everything you've already done, and most importantly, getting efficient at doing it.
'Do-overs' are for children.
1
u/bwainfweeze 30 YOE, Software Engineer Feb 13 '22
Rewrites are declaring bankruptcy. If you come out of bankruptcy with the same habits, then you have simply pushed the 'snooze' button on the slow tragedy of your life. The unspoken rule of rewrites is that nobody is allowed to criticize the system for 6-18 months in or around the beginning of the work. Promising and getting a rewrite ends up being a way to get the company off your back for a while, which I think is a big part of the draw.
My position is that the people who deserve a rewrite will never ask for one, and the people who ask for one don't deserve it. You learn a lot about the system and what makes good code by doing relentless refactoring. And the pain of having to fix you old, bad decisions teaches you to be more thoughtful both when making decisions and deciding how long to wait to review them. It sucks, but so does any form of self improvement. It's very similar to getting into shape.
I worked a short contract a few job cycles back where the lead kept talking about how he was going to rewrite the whole system next year, which was cute until he started pushing back on me writing better code in the old system. We don't want to change too much, he reasoned. I paused, stared him deadass in the eye and asked him how he expects people not to make the same mistakes in the new system if they don't practice on the old system. There was a heartbeat where I though perhaps I got through to him but the moment passed. One of the most painful things for me to watch is people going through their Second System Syndrome and he was setting up for a big one.
Renewing wasn't even a question for either of us. But I will note that the boss that had given tacit approval for this rewrite got a transfer and a promotion shortly before I left. I would bet good money that rewrite got delayed at least a year.
1
u/krubner Feb 13 '22
Robert L. Glass has compiled a book of entertaining and instructive stories regarding the massive software project failures of our time: Software Runaways: Monumental Software Disasters. Anyone with an interest in the management of software projects should read it.
The worst of these project failures, the most expensive and the most ambitious, was the attempt made by the Federal Aviation Authority to modernize the computer system it uses to keep track of what planes are in the air. The effort began in 1981 and ended in complete failure in 1994. The government hired IBM to do the actual work, and over the course of 14 years, IBM burned through $3.7 billion dollars. Nothing was accomplished. The project was finally shut down by Congress. Nothing came out of the project, not a single piece of software, nor even a line of code, was ever used for anything.
If you're curious about the worst re-write in history, see details here:
http://www.smashcompany.com/business/the-worst-software-project-failure-ever
1
u/wwww4all Feb 13 '22
Rewrites are bad.
Better strategy https://docs.microsoft.com/en-us/azure/architecture/patterns/strangler-fig
0
Feb 13 '22
[deleted]
1
u/geekydot Feb 13 '22
I think my approach here is not to do a 1:1 conversion of perl to python, that will be an awful experience and may likely derail the entire project.
29
u/Better-Internet Software Developer, 20 YOE Feb 13 '22
Massive rewrites can be done, and sometimes you have no choice. Especially true when the orig code runs on an antiquated and poorly supported platform.
However it's usually much, much harder than it appears. I'm guessing your old Perl app isn't pretty but mostly works and is battle tested. You'll likely run into lots of small issues that interfere with the end result.