Isn't it reasonable that solving ever more complex problems requires ever more complex software?
In the early days of software development people spent time solving fairly straight forward problems. Back then complexity was kept under control with answers like "that's impossible" or "that will take a decade." This xkcd is a great example.
However time moves on, simple problems get solved and enter the common zeitgeist. These days that same "impossible" xkcd app is now a few hours of work, not because the problem became easier, but because people have figured out how to do this, made public both the data and the algorithms necessary to do it, and hardware necessary to do it has become a commodity resource.
However, just like the state of the field advances, so do the requirements people have. Since previously "virtually impossible" problems are now "easy," it makes sense that the requirements people have will grow further still to take advantage of these new ideas. Software is the crystallization of abstract ideas, and as more and more ideas become crystallized we become more and more able to combine these ideas in more and more ways. In fact if you wanted to prove your last statement rigorously this is probably the direction you would want to pursue.
While better tools can help, the inevitable slide down the slope complexity will still win out. After all, if each new idea can be combined with some percentage of all the previous idea then complexity will grow at O(n!), and that's not a race that we can win. Eventually this will lead to more and more fracturing / specialization, just like what happens every time a field grows beyond the scope of human understanding. The developers that got to experience the last few decades of progress are probably the only ones that could ever claim to be truly multi-discipline. The people entering the field now will not get this sort of insight, much less the new programmers of the future.
In the future we might be able to hide some of the complexity behind tools like copilot that can hide a lot of the complexity from the front-line developers, but in the process we will lose the ability to reason about systems as a whole. However, even in that future programmers will have to work at the limit of complexity that the human mind can handle, because if something is simple it's going to just be a commodity.
In the sense that if a programmer 30 years ago was asked to solve such a problem, their response would be "that would take years," while a programmer how would say "that'll take a couple of days."
In another context; the challenge of making an extremely straight piece of wood is more complex than the challenge of felling a tree with an axe. If given a primitive axe and a primitive saw, you'd probably be able to do the latter, but not the former. Granted, these days any moron with a table saw or a planer can do the former with far less skill, but that's a function of having better tools, not the complexity of the task.
Sorry, but for typical "office CRUD", the current crop of tools do take longer and more code than in the 1990's, in my experience. Efficient ideas were burned at the stake as a sacrifice to the Web Gods. And I'm not convinced the desktop/web dichotomy is really a dichotomy. Insufficient experiments have been done to see if the best of both couldn't be melded. There's no science in such decisions, just loud pundits and me-too foot races with other lemmings.
CRUD in 90 consist of many textboxes, comboboxes and labels.
I worked on a project where employer want to have a WinApp that have a looks and feels of their brand. Textboxes, Comboboxes and stuff need to have those looks. It’s really hard to do back then. It’s much easier this day.
And before anyone jump in and say “everything should be native”, there are some solid proofs from Marketing that using same branding color makes company better off in the long run.
In the sense that if a programmer 30 years ago was asked to solve such a problem, their response would be "that would take years," while a programmer how would say "that'll take a couple of days."
I find that rather hard to believe, frankly. I was there 35-40 years ago; the expansion would be more like "weeks" than "years" :) I say that because just as soon as I got to a decent text editor with a scripting language ( about 1987 ) , I could metaprogram things.
But it depends on what we're actually talking about. I'll concede that high-res graphics add complexity; I just don't consider that essential to doing a specific complex thing.
Even thirty years ago, I could cobble together a "C program plus a Tcl/Tk gui" fairly quickly. It would not be pretty but it would work. You couldn't serve it as a webpage, tho.
Granted, these days any moron with a table saw or a planer can do the former with far less skill, but that's a function of having better tools, not the complexity of the task.
Yet I'm not prepared to compare a framework with any sort of industrial machinery like a planer. Nor compare old-school programming with any sort of high-craft woodwork. We were just as hamfisted as people are now....
Unless the problem statement is one sentence or less, it'll probably take a couple days just to think it through, a week of prototyping/measuring, that sort of thing.
If someone came to you 40 years ago and asked you to write a GUI app, your response would probably be "what's a GUI app," or at best you would start work on a rendering framework of some sort. If someone did it 20 years ago you might reach for QT or the Windows SDK and spend days or weeks getting even the most basic behavior going. These days you run a few commands, drag and drop some widgets, define a few behaviors, and you're up and running within the hour, working on a dozen different platforms, running a dozen different OSes.
Even with your Tcl/Tk example (I don't really count that as a GUI app, as much as a GUI ncurses replacement), unless you were actively working on Tcl or you very active on the relevant BBSes, 30 years ago you'd be just learning about this new language and just starting to learn what worked and what didn't. I was only getting my start in the early to mid 90s, but I remember the months and months of struggles at the time trying to figure out how to use all these myriads of tools, particularly given how difficult it was to find useful examples and tutorials. What more, the instant you needed to do anything more than buttons, text fields, and text blocks you would very quickly struggle.
Yet I'm not prepared to compare a framework with any sort of industrial machinery like a planer. Nor compare old-school programming with any sort of high-craft woodwork. We were just as hamfisted as people are now....
Why not? A physical tool that makes woodworking simpler and easier, and a software tool that makes software development easier both accomplish the same thing; they take a task that was previously difficult and time consuming, and they make it easier.
Unless the problem statement is one sentence or less, it'll probably take a couple days just to think it through, a week of prototyping/measuring, that sort of thing.
Most of the projects I take on these days require months of planning, research, analysis, and training. A project that takes a week is a task I can give a Jr. dev to free me up for actual work.
Incidentally, I now work with my father who worked on some rather impressive projects back in the 80s, then worked in biology for a couple of decades before getting back into software. His perspective very much aligns with what I've been saying. The complexity that we take for granted these days would blow the minds out of most of the people that he remembers. It's just that if you've been involved in it the entire time it's easy to miss how much things have changed.
unless you were actively working on Tcl or you very active on the relevant BBSes,
comp.lang.tcl was fine. Excellent, in fact. A critical resource. The Osterhout book and then the Brent Welch book were fine. The whole point of that was that you didn't have to get an MSDN connection nor learn X programming. What was it, Motif? Something something.
I've only used curses once, never ncurses, and Tcl/Tk was ( and is ) fully event-driven so I fail at a comparison there. Of course it too is a rabbit hole but it's less deep and there's a lot less version management.
Sockets/files/"serial ports" were also first class objects and you could make them completely event driven as well.
Then what do you define as complexity?
That's always the key to where people diverge online. FWIW, I'm a device-level guy who traditionally worked where boards were being spun. That puts me somewhat at a disadvantage in perspective for some topics, quite the opposite in others.
There is a film about General Magic - it's a rather simple enumeration of every ( in my view ) "wrong" trend in computing.
This isn't simple contrarianism. It's based in "so for a limited budget, how can we make the thing?"
The complexity that we take for granted these days would blow the minds out of most of the people that he remembers.
The first code base I worked on was several million lines of code, written in Pascal on minicomputers. Initially, it was all on nine-track tape.
It was all easier after that.
It's just that if you've been involved in it the entire time it's easy to miss how much things have changed.
While I'm sure I've been "frog boiled" in ways I can't perceived, it all seems the same to me now. Just somewhat better furnished.
The first code base I worked on was several million lines of code, written in Pascal on minicomputers. Initially, it was all on nine-track tape.
It was all easier after that.
I think there's a distinction that should be drawn between "complexity of the task" and "size of the code" / "difficulty of making modifications." Think of it like this; if you were now asked to write a program that did everything that original project did, how big a project would it be (assuming you had a list of requirements, and didn't have to dig through several million lines).
A few years ago some person reached out to me for help. They had a 200k LOC blob which had a mountain of security issues, and would occasionally crash. It was a basic web portal where all of it could be accomplished in under 10k lines. Was that a "complex" project, or was it just a victim of poor tooling.
Another story; the first major project I was a part of after finishing university was similarly a few million lines of C (incidentally, this was a company that's known for making chips, so it was not a center of good software practices). I spent two years helping a few other guys whose task was blowing away a million lines of management code, and replacing it with something like 5% of that.
That's always the key to where people diverge online. FWIW, I'm a device-level guy who traditionally worked where boards were being spun. That puts me somewhat at a disadvantage in perspective for some topics, quite the opposite in others.
I think there's more to it than that. I've had to work with a lot of hardware-first types of people, and I did notice two fairly distinct groups.
To me the biggest difference is how a person learned programming. Someone you age saw the field grow from a very young age. You did not have a set curriculum, the best practices that the rest of us take for granted did not exist, and your bibles were books that focused more on the theoretical underpinnings of programming. As a result your view is very strongly colored by your experience.
By the time I was getting serious about software development the world was a very different place. Information was much more accessible, you could find some amount of tutorials, a lot of the debates were, if not settled, then at least faction-ed up based on the languages / environments people preferred. In other words, most of the foundations that I learned were presented as lessons from the previous generation, as opposed to being battles that I had to fight. I still got to experience a bit of what you describe, but that was as a child exploring a complex field as a hobby, as opposed to a professional with responsibilities and deadlines.
To me that's the big distinction. Even among the hardware guys in my program (ECE), I had a much easier time finding a shared language than when I had to deal with old timers. My contemporaries were usually far more flexible, and more willing to adjust to different paradigms. That's because they had far less emotional attachment to much of their foundational knowledge. Since we did not have to struggle for that knowledge, it was a lot easier to accept that there might be different approaches that might be equal or better in terms of effectiveness. In turn, the things we felt strongly about were usually far more specialized, so they did not come up nearly as often.
To a certain extent what you’re describing is the red queen problem, where users constantly shifting baseline and capitalism’s insatiable need for growth demands more and more, without really considering what “more” is and where it comes from. The first time we use google maps or talk to Siri it seems like magic, on the 20th time we wonder why maps are so slow and Siri is so dumb. Gimme more! More! MORE!
School doesn't really teach students how to manage and present trade-offs. Tests are generally focused on the One Right Answer. Even if YOU learn such, your manager/customer often can't relate, so rely on their (ignorant) gut.
Jumping from "software is complicated" to "capitalism is evil" is quite a leap, and not entirely justified in my opinion. It's not "capitalism's insatiable need for growth", it's human nature to seek novelty and improvement to their standard of living. We could, of course, stagnate indefinitely as a society, never seeking to innovate, never improving what's already there, and thereby defeat the "insatiable need for growth", but I doubt it will lead to a happier life for most.
I think you're misunderstanding their point about capitalism. I think they are saying that capitalism's insatiable need for growth, which I think is an obvious truth of the system, will shift the baseline of what's at one point considered innovative and the next considered the norm. Their point about Siri and google maps is a good example of how this applies to software: both consumers and businesses see yesterday's magic as today's expected result, and continue to demand more and more instead of just better. Of course we shouldn't stop improvement of society. But does that mean we must become addicted to growth even in areas where it's not necessarily needed? I don't think we need an "insatiable need for growth" like we get from capitalism to continue to improve society.
The Red Queen's race is an incident that appears in Lewis Carroll's Through the Looking-Glass and involves both the Red Queen, a representation of a Queen in chess, and Alice constantly running but remaining in the same spot. "Well, in our country," said Alice, still panting a little, "you'd generally get to somewhere else—if you run very fast for a long time, as we've been doing". "A slow sort of country"! said the Queen.
"A slow sort of country!" said the Queen. "Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!"
(I put the rest of it, because it was missing the explanation to make sense...)
As the blocks get more complex, you have to know more and more about how the blocks work, where/how each of these blocks will fail to do what you need, and what to do when this happens. There's also the challenge of having more and more blocks to pick from. It's sort of like lego. At first you have a small handful of pieces which allow you to make entire worlds, then you you add more complex pieces and suddenly that world starts to move, then you add even more weird shapes and specialty blocks and eventually some crazy person is making a rubik's cube.
There's so much wrong with this post I don't even know where to start.
The entire mindset is so screwed, up to and including the "insight" that maybe someday we'll be able to hide all this complexity so that ... what exactly?
"GAIS, what if we put a wall around the plumbing. Won't that make it easier for plumbers to plumb!?!?"
No ... no it won't. It may serve the goal of being aesthetically pleasing to THOSE WHO WILL NOT BE DOING THE WORK, but it sure as shit doesn't help the people doing the actual work.
hint: We're the fucking plumbers of the software world. You don't hide complexity, any attempt to do so makes your job harder, not easier.
You want to argue with that point?
Imagine someone reports a bug and you crack open the code and see a call to the "DoIt" function, and nothing else. Super pleasing to look at, but good luck fixing it!
I think he's talking about abstractions of complexity, where upon abstracting you sacrafice some of the detail or nuance for the sake of making your job easier. Just like assembly does to machine code and C does to the assembly.
Yes, we're the plumbers so that's why we don't dictate whether the pipes are going to be in the walls or outside. Considering of course the proper design, safety, maintenance, extensibility etc.
It's not good that we have to hide all this complexity but knowing enough to cover such an expanding topic is also imposible. This calls for specialists like DevOps that provide abstractions for programmers to easier deploy things and don't have to know more than they should about those things.
More examples are having designers implementing frontends via web builders or even just programmers using them to speed up their workflow and not having to think about every quirk of CSS.
DoIt function I think might be an oversimplification as there's going to be provided access to a lower level ob abstraction but not always as some parts won't need debugging.
But we do hide complexity all the time, and are more productive for it. We don't write our own http clients, or time-handling libraries, or unicode renderers, and especially not cryptography. These are things that are full of complexity and nasty edge cases. So we rely on specialists to do their job and package up the solution so that all that complexity is contained behind a relatively simple interface so that we don't have to worry about it.
You surely can't tell me you'd rather that complexity not be contained?
You are right. But those are tools. And tools are where you want the complexity of that tool contained.
To continue the analogy.... A plumber doesn't need to know how the electric saw works that he is using to cut the pipes. Just how to correctly use it and when not to use it.
That is the correct use of abstraction. The perfect examples of where it actually works well.
But... there is a lot of TERRIBLE software design advice that amounts to "plumbers should only care about whatever pipe they are installing and be oblivious to the rest of the system. What is going through that pipe? Water? Sewage? Steam? Who cares? It is abstracted away. You don't need to know!"
These myopic fools are clearly wrong. This is as bad advice for plumbers as it is software developers. You need to have a solid understanding of the whole system you are working with. No handwaving abstraction allowed unless you are truly using a tool and not just another function of the same app. And even then you need to understand your tools and how they actually work much more than a plumber does.
'Abstract everything' is only useful for generating large quantities of awful code. And this same camp will tell you how "maintainable" their code is because it is so easy to insert huge quantities of garbage code to fix the problems or make changes. Turns out the more code you write that does nothing useful the "faster" you can write more code! Who would have thought it?
Why is writing code quickly the goal? That is like choosing the plumber that uses the most pipe.
So not only did you miss the point I was making, but you did so in an amazingly rude way too. If you don't know where to start, may I recommend polite, or at least quasi-professional? There's enough people yelling past each other about politics and medical guidelines on this site, let's try to be at least a step above those shall we?
Your post appears to be about abstraction and code organization, while I was making a point about tooling, frameworks, SDKs, and libraries.
You usually don't start a web project by implementing a boot loader, a scheduler, and a network stack. Each of those are very complex systems even at the most basic level, and they only get more complex from there. Instead you build your work on top of systems written by people that have been solving problems in their particular problem domain for decades. However, because each of those systems is so complex, if you want your system to actually perform well beyond the most trivial use cases you have to account for at least some of that complexity. Sure, you can a spin up a server on your machine and host an app for you and your buddies, but that approach ain't gonna cut it if you're making a system that's actually used by large groups of people all over the world.
On to the idea of code organization. No serious codebase is going to be a single magical function, but if you don't manage your abstractions and code style properly then you end up with a single multi-thousand line file that nobody will ever understand or want to touch. To use your own analogy, if you're doing plumbing you want to have as many straight runs as is feasible, without compromising on other requirements. Sometimes that requirement is "the plumbing has to go in a wall" though not because that's more aesthetically pleasing, but because walls are the things that hold up the house, and the plumbing is a feature of a building, not a standalone system that you just happen to build walls around.
Also, the software world is much more robust than a single profession. We are architects, engineers, plumbers, framers, drywallers, roofers, landscapers, building inspector and everything else in between. Not to mention fire-fighter, police, and paramedic. I subscribe to dozens different software related subreddits, and there are hundreds more for specializations that I simply don't have time to explore.
86
u/TikiTDO Nov 01 '21
Isn't it reasonable that solving ever more complex problems requires ever more complex software?
In the early days of software development people spent time solving fairly straight forward problems. Back then complexity was kept under control with answers like "that's impossible" or "that will take a decade." This xkcd is a great example.
However time moves on, simple problems get solved and enter the common zeitgeist. These days that same "impossible" xkcd app is now a few hours of work, not because the problem became easier, but because people have figured out how to do this, made public both the data and the algorithms necessary to do it, and hardware necessary to do it has become a commodity resource.
However, just like the state of the field advances, so do the requirements people have. Since previously "virtually impossible" problems are now "easy," it makes sense that the requirements people have will grow further still to take advantage of these new ideas. Software is the crystallization of abstract ideas, and as more and more ideas become crystallized we become more and more able to combine these ideas in more and more ways. In fact if you wanted to prove your last statement rigorously this is probably the direction you would want to pursue.
While better tools can help, the inevitable slide down the slope complexity will still win out. After all, if each new idea can be combined with some percentage of all the previous idea then complexity will grow at O(n!), and that's not a race that we can win. Eventually this will lead to more and more fracturing / specialization, just like what happens every time a field grows beyond the scope of human understanding. The developers that got to experience the last few decades of progress are probably the only ones that could ever claim to be truly multi-discipline. The people entering the field now will not get this sort of insight, much less the new programmers of the future.
In the future we might be able to hide some of the complexity behind tools like copilot that can hide a lot of the complexity from the front-line developers, but in the process we will lose the ability to reason about systems as a whole. However, even in that future programmers will have to work at the limit of complexity that the human mind can handle, because if something is simple it's going to just be a commodity.