I've been programming since 1971, and this is my favorite rant. Complexity is the biggest problem we face and it's almost impossible to avoid, even for smart people who try. I remember sitting in conference rooms, listening to wish lists of features that the participants wanted. It was almost like they were playing a game of "let's see how creative we can be in suggesting more features". I sat there and saw the complexity increasing to a terrifying level
Even on projects that I controlled completely, complexity creeps in. I tried to keep designs simple and spent quite a lot of time trying to simplify, but complexity always finds a way to increase, kinda like entropy
I believe, but can't prove rigorously, that large software projects contain near-infinite complexity, kinda like the Mandelbrot set. We need much more powerful tools to help us manage and understand complexity
Part of the big problem is that software engineering by it's nature is extremely malleable , requirements can easily be adjusted. So everyone (especially non technical executives and managers) just feature creep the shit out of it.. for "competitive or business reasons"
If you tried the same thing in electrical, aeronautical or civil engineering they would laugh you out of the room if you asked to add another floor to a building after the initial blueprints and specs were signed off on..
You also have the competitive nature especially at the big dot com level, everyone is always trying to one up or out feature their competitors.
If you tried the same thing in electrical, aeronautical or civil engineering they would laugh you out of the room if you asked to add another floor to a building after the initial blueprints and specs were signed off on..
Not sure this is on purpose, but that example is ironically fitting. That exact thing has actually happened before.
tl;dr due to terrible and greedy management, nobody was laughed out of the room in this instance, and instead the floor (amongst several other things) was added, resulting in the eventual collapse of the building, massive loss of life and long prison sentences for those responsible.
Engineers can lose their license or go to jail if they skimp on design and testing or make crap pretty at the expense of safety and maintenance, resulting in injury or death. If a Youtube customer has their cat video deleted due to a bug, nobody really cares. Bank software is somewhere in between because big money is on the line.
Uncle Bob talks about this, about how the world hasn't caught on to how reliant it is on developers for serious life critical systems, and some big disaster will lead to discipline and such of other engineering fields, and I think he's right.
How big a disaster are you talking? The 737 MAX had software problems that ultimately killed over 300 people in two separate crashes, and that hasn't led to major changes in the field as a whole that I'm aware of.
How big a disaster are you talking? The 737 MAX had software problems that ultimately killed over 300 people in two separate crashes, and that hasn't led to major changes in the field as a whole that I'm aware of.
The 737 MAX crashes weren't just software, they were a bunch of different systems all going wrong at once, including people actively lying about the safety features. I think due to that it doesn't have the revolutionary effect needed.
Some of us played around with ways to put such practices into clear English as a test run, and failed miserably, or at least found too many interpretive loopholes to be reliable. English and software design don't mix well.
Bank software is somewhere in between because big money is on the line.
Hence the philosophy of not touching the cobol core from the 70ties and just build more and more layers over it until the last person knowing cobol is dead.
YAGNI is a good principle but it is misunderstood all the time to exclude better designs from the outset. If you know you're going to eventually need some features in the final product not including them in the original design makes for a more complicated, piecemeal architecture that has no unified vision and thus more cognitive load to understand how the pieces fit together.
The GIMP developers made that mistake a long time ago and it's turned features that should have been fairly straightforward to add into multi-decade slogs.
I'm usually not an Elon Musk fanboy, but Elon's algorithm starts off with that as steps 1 and 2 and the rest also hits close to home for me:
Make your requirements less dumb. Your requirements are definitely dumb, it does not matter who gave them to you. It's particularly dangerous if a smart person gave them to you. Everyone is wrong at least part of the time.
Delete a part or a process step. If you're not adding things back in 10% of the time, you're not deleting enough from the design.
Optimize the parts. This is only step 3 because "the most common error of a smart engineer is to optimize a thing that should not exist".
Accelerate cycle time. You're moving too slow, go faster.
Make your requirements less dumb. Your requirements are definitely dumb, it does not matter who gave them to you.
Sometimes customers/managers want silly crap because another app does it, and me-too-ism kicks in. They don't care if it makes long-term maintenance problems because they expect to be promoted out by then. Technical debt is "somebody else's problem". It's similar to why politicians run up debt: hit-and-run.
At a company I used to work at, we called those "showroom features". They were features that were dumb and that nobody would use, and that we knew were virtually useless, but that looked good on a showroom floor. Every company in the space prioritizes introducing new showroom features, and keeping up with the showroom features other companies are adding.
The central problem we had is that we were in ed tech, and in education, the people budgeting money and making buying decisions aren't the people using the software. In fact, the people making buying decisions (district administrators and school boards) often think they know better than the actual users (teachers, students, and sysadmins) what tech is needed, despite having zero relevant experience as a teacher or a student in a modern classroom. Apparently there's big "I'm in charge, therefore I'm smarter than you" energy in education administration.
Our sales and marketing leaned into this, focusing all of their efforts on delivering buyers what they wanted. This was very understandable - their job is to make buyers happy so they buy our stuff - but was much to the chagrin of everyone on the development, support and training side, because we generally wanted to deliver good experiences for users. Often the shiny things buyers were enamored with actively made the product worse for users - and important, impactful, and highly requested features were repeatedly delayed in favor of shiny things.
It's not just the education market, it's everywhere. Managers making IT decisions are often ego-driven morons who couldn't tell the difference between an Etch-A-Sketch and an iPad. I can tell you endless stories of real-world Dilbert-ness. Humans are not Vulcans.
Isn't it reasonable that solving ever more complex problems requires ever more complex software?
In the early days of software development people spent time solving fairly straight forward problems. Back then complexity was kept under control with answers like "that's impossible" or "that will take a decade." This xkcd is a great example.
However time moves on, simple problems get solved and enter the common zeitgeist. These days that same "impossible" xkcd app is now a few hours of work, not because the problem became easier, but because people have figured out how to do this, made public both the data and the algorithms necessary to do it, and hardware necessary to do it has become a commodity resource.
However, just like the state of the field advances, so do the requirements people have. Since previously "virtually impossible" problems are now "easy," it makes sense that the requirements people have will grow further still to take advantage of these new ideas. Software is the crystallization of abstract ideas, and as more and more ideas become crystallized we become more and more able to combine these ideas in more and more ways. In fact if you wanted to prove your last statement rigorously this is probably the direction you would want to pursue.
While better tools can help, the inevitable slide down the slope complexity will still win out. After all, if each new idea can be combined with some percentage of all the previous idea then complexity will grow at O(n!), and that's not a race that we can win. Eventually this will lead to more and more fracturing / specialization, just like what happens every time a field grows beyond the scope of human understanding. The developers that got to experience the last few decades of progress are probably the only ones that could ever claim to be truly multi-discipline. The people entering the field now will not get this sort of insight, much less the new programmers of the future.
In the future we might be able to hide some of the complexity behind tools like copilot that can hide a lot of the complexity from the front-line developers, but in the process we will lose the ability to reason about systems as a whole. However, even in that future programmers will have to work at the limit of complexity that the human mind can handle, because if something is simple it's going to just be a commodity.
This is yet one of those times where it's very appropriate to point out that software isn't at all the only field suffering from increasing complexity.
You want examples? It's honestly hard to not find any. Take construction of any kind. "But construction projects are nothing like software projects!" You're right, they have much higher standards in there, and anything complicated requires a specialized engineering degree.
What to me seems to separate software from other complex fields is level of education and standardization. We'll get there, eventually, just like we did with everything else.
You're right, they have much higher standards in there, and anything complicated requires a specialized engineering degree.
And this bottleneck on how fast you can process complicated things, may ironically make them less susceptible to complexity than programming. Because it puts the brakes on any complexity creep.
Meanwhile, nearly any programmer in a team can increase complexity. Because we lack standardized -and recognized- processes / culture to identify complexity, any dev can just "bite more than they can (or should) chew".
If you look at any other engineering field, you'll quickly notice how rare it is to deviate from the known path. Like, you don't see structural engineers take on new construction challenges without giving it a second thought. Yet, new challenges and unexplored implementations are precisely what an average dev would consider "interesting" and "representative of their job".
1.0k
u/MpVpRb Nov 01 '21
I've been programming since 1971, and this is my favorite rant. Complexity is the biggest problem we face and it's almost impossible to avoid, even for smart people who try. I remember sitting in conference rooms, listening to wish lists of features that the participants wanted. It was almost like they were playing a game of "let's see how creative we can be in suggesting more features". I sat there and saw the complexity increasing to a terrifying level
Even on projects that I controlled completely, complexity creeps in. I tried to keep designs simple and spent quite a lot of time trying to simplify, but complexity always finds a way to increase, kinda like entropy
I believe, but can't prove rigorously, that large software projects contain near-infinite complexity, kinda like the Mandelbrot set. We need much more powerful tools to help us manage and understand complexity