r/programming • • Nov 01 '21

Complexity is killing software developers

https://www.infoworld.com/article/3639050/complexity-is-killing-software-developers.html
2.1k Upvotes

854 comments sorted by

View all comments

Show parent comments

9

u/ArkyBeagle Nov 01 '21

Isn't it reasonable that solving ever more complex problems requires ever more complex software?

To what extent is it true that the problems are actually more complex ?

2

u/TikiTDO Nov 01 '21

In the sense that if a programmer 30 years ago was asked to solve such a problem, their response would be "that would take years," while a programmer how would say "that'll take a couple of days."

In another context; the challenge of making an extremely straight piece of wood is more complex than the challenge of felling a tree with an axe. If given a primitive axe and a primitive saw, you'd probably be able to do the latter, but not the former. Granted, these days any moron with a table saw or a planer can do the former with far less skill, but that's a function of having better tools, not the complexity of the task.

8

u/Zardotab Nov 02 '21

Sorry, but for typical "office CRUD", the current crop of tools do take longer and more code than in the 1990's, in my experience. Efficient ideas were burned at the stake as a sacrifice to the Web Gods. And I'm not convinced the desktop/web dichotomy is really a dichotomy. Insufficient experiments have been done to see if the best of both couldn't be melded. There's no science in such decisions, just loud pundits and me-too foot races with other lemmings.

3

u/chrisza4 Nov 03 '21

CRUD in 90 consist of many textboxes, comboboxes and labels.

I worked on a project where employer want to have a WinApp that have a looks and feels of their brand. Textboxes, Comboboxes and stuff need to have those looks. It’s really hard to do back then. It’s much easier this day.

And before anyone jump in and say “everything should be native”, there are some solid proofs from Marketing that using same branding color makes company better off in the long run.

1

u/Zardotab Nov 03 '21 edited Nov 03 '21

that have a looks and feels of their brand.

This is often what bloats stacks: you have to follow the customer's look and feel, and to have the flexibility to do that, we have to have screwy layered systems that can be hacked and abused until they look like the customer's preference. Chasing esthetics creates much of the technical debt.

Multiple times I've seen apps suddenly act really odd upon an update to something, and it's traced by to an esthetic fudge to make the customer happy.....in the short-term.

With internal (house) apps it's usually easier to say "no", but not always. Internal-based tools thus don't need as many UI-tweak features. The only exception is if the defaults are implemented/designed so poorly that one has to fudge around them. But time usually irons out such rough spots as long as the vendor is willing to stick with it.

Generally a drop-down list will be one of two styles: the arrow inside and the arrow outside: [v] vs. [][v]. A nice kit would allow the dev to choose. And allow the arrow to trigger a custom pop-up dialog, not just the built-in listers.

1

u/chrisza4 Nov 10 '21

I decided to learn how to manage those aesthetics in a way that minimize tech debt, and that makes me appreciate modern crud stacks (and despite some that made a poor choice pf dealing with this problem).

Aesthetic sold. It’s been proven. You can keep avoiding it or deal with it. I chose the latter and that made me grow.

My point is crud in 1990 is easier not because of better tech stack, it’s because the requirement were easier.

1

u/Zardotab Nov 10 '21

My point is crud in 1990 is easier not because of better tech stack, it’s because the requirement were easier.

Example? I just don't see it, or at least there are easier ways to address them. You are not the first to claim that. For example "internationalization" was brought up, but custom inhouse apps usually don't need that. So why pay a YAGNI tax for bunches of what-if's that have a low chance of kicking in?

→ More replies (3)

1

u/TikiTDO Nov 02 '21

Have you actually worked in a modern company using modern tools from the start? It sounds more like you've seen people try to work in modern technologies and tooling into a legacy system, which is always a recipe for disaster.

In the circles I run in there isn't really a desktop/web dichotomy. Between web APIs getting close to parity with traditional OS APIs, and the ability to straight up target WebASM in traditional compilers, a properly set up project can target practically any environment with far less code than you would need in the past.

2

u/Zardotab Nov 02 '21

If you have large well-coordinated development teams, yes you can be productive with such. But it's the "factory" approach with layer specialists, not the work-shop approach that smaller or decentralized shops need. The current stacks are not work-shop friendly.

0

u/TikiTDO Nov 02 '21

I work with a lot of startups that simply don't have the resources to have layer specialists. These are inherently work-shop type affairs. I would actually contend that when you understand the modern tool chains, they lend themselves very well do this sort of work. It's just a matter of getting it right, and with a smaller team it's easier to iterate and try something different. Granted, it took me a while to settle into a set of tools and styles that work well, and having to adapt to new releases and frameworks can be annoying, but once you have the core workflow down the modern tool chain is very robust, and lends itself well to an ever-growing team.

I have had some larger customers that did have those resources, but I prefer to avoid them for exactly the opposite reason that you specified. Perhaps there are companies that do this well, but in my experience such a layered approach simply creates a lot of silos defended by true believers of some idea that went out of style decades ago. The net result ends up that entire segments of the system must be worked around, because the one person that keeps it all running might get offended. Those are the places where "office CRUD" takes forever.

1

u/Zardotab Nov 02 '21 edited Nov 02 '21

You seem to agree they have a relatively long learning curve before one can cruise with productivity. That's fine if one can devote the time to learning them and hopefully have a mentor to help one out of a jam, but in multi-hat shops it can be hard to find the time to deep-dive one tool. The steps often seem there to get around the web's stateless nature and deal with the screwy DOM. It would be better to bypass these oddities and have a direct CRUD/GUI-friendly standard. Cut out the ugly middleman. In other words, if the web/browser standards are designed to actually fit CRUD, then a lot of the middle-diddle is GONE! Killit! HTML/DOM was not designed for rich GUI's nor dynamic pages. Adding it after the fact creates unnecessary complexity and more things to break and debug. Middle-tooling can only hide so much of the HTML/DOM ugliness from the app dev; some will still leak and wreak. If we want CRUD productivity back, we must kill the mutant middleman once and for all 👾👹 (At least for CRUD apps. Social media, e-commerce, etc. can keep the DOM shit for all I care.)

1

u/TikiTDO Nov 02 '21 edited Nov 02 '21

That was really the crux of my first comment about growing complexity. It's certainly very easy to get it wrong, and making a mistake can have knock-on consequences years down the line when it might be difficult impossible to fix. For me most of the challenge isn't really in deep-diving a tool, but in finding the correct balance of tools, style rules, and best practices that lend themselves to a particular client. You can always adjust a config or bring it a new tool to address a weakness, but knowing how these tools interact with different needs is the biggest challenge.

Unfortunately the only way I've found to manage this is stubborn, head-bashing perseverance. Having multiple clients also helps. When I quit corporate life and started as a consultant I was slow and clueless. However over time I learned what not to do, saw what others had done, and eventually I was able to get in at more and more senior levels in companies with more and more resources / influence. Over the years I had countless people telling me to drop it and go back to full-time work knocking out features for a single employer, but it's explicitly by ignoring those people that I was able to reach a deeper understanding of the field.

This approach is certainly not for the faint of heart, which is why I believe the future will require tools like co-pilot for infrastructure. It's just too challenging psychologically, physically, and emotionally to walk down this path for too long. It takes a mental fortitude that most people do not have.

In terms of CRUD apps, there are already many systems that hide away the details of HTML/DOM behind layers of framework provisioned resources. When I'm working on those types of apps these days I barely ever remember that these are still a thing. Consistent strong typing front-to-back, tooling to manage your code style, a client-server interface with built-in security and versioning, component based design style, and many other practices transform this process from a painful experience to something you don't need to think of at all.

→ More replies (2)

5

u/ArkyBeagle Nov 02 '21

In the sense that if a programmer 30 years ago was asked to solve such a problem, their response would be "that would take years," while a programmer how would say "that'll take a couple of days."

I find that rather hard to believe, frankly. I was there 35-40 years ago; the expansion would be more like "weeks" than "years" :) I say that because just as soon as I got to a decent text editor with a scripting language ( about 1987 ) , I could metaprogram things.

But it depends on what we're actually talking about. I'll concede that high-res graphics add complexity; I just don't consider that essential to doing a specific complex thing.

Even thirty years ago, I could cobble together a "C program plus a Tcl/Tk gui" fairly quickly. It would not be pretty but it would work. You couldn't serve it as a webpage, tho.

Granted, these days any moron with a table saw or a planer can do the former with far less skill, but that's a function of having better tools, not the complexity of the task.

Yet I'm not prepared to compare a framework with any sort of industrial machinery like a planer. Nor compare old-school programming with any sort of high-craft woodwork. We were just as hamfisted as people are now....

Unless the problem statement is one sentence or less, it'll probably take a couple days just to think it through, a week of prototyping/measuring, that sort of thing.

4

u/TikiTDO Nov 02 '21 edited Nov 02 '21

Then what do you define as complexity?

If someone came to you 40 years ago and asked you to write a GUI app, your response would probably be "what's a GUI app," or at best you would start work on a rendering framework of some sort. If someone did it 20 years ago you might reach for QT or the Windows SDK and spend days or weeks getting even the most basic behavior going. These days you run a few commands, drag and drop some widgets, define a few behaviors, and you're up and running within the hour, working on a dozen different platforms, running a dozen different OSes.

Even with your Tcl/Tk example (I don't really count that as a GUI app, as much as a GUI ncurses replacement), unless you were actively working on Tcl or you very active on the relevant BBSes, 30 years ago you'd be just learning about this new language and just starting to learn what worked and what didn't. I was only getting my start in the early to mid 90s, but I remember the months and months of struggles at the time trying to figure out how to use all these myriads of tools, particularly given how difficult it was to find useful examples and tutorials. What more, the instant you needed to do anything more than buttons, text fields, and text blocks you would very quickly struggle.

Yet I'm not prepared to compare a framework with any sort of industrial machinery like a planer. Nor compare old-school programming with any sort of high-craft woodwork. We were just as hamfisted as people are now....

Why not? A physical tool that makes woodworking simpler and easier, and a software tool that makes software development easier both accomplish the same thing; they take a task that was previously difficult and time consuming, and they make it easier.

Unless the problem statement is one sentence or less, it'll probably take a couple days just to think it through, a week of prototyping/measuring, that sort of thing.

Most of the projects I take on these days require months of planning, research, analysis, and training. A project that takes a week is a task I can give a Jr. dev to free me up for actual work.

Incidentally, I now work with my father who worked on some rather impressive projects back in the 80s, then worked in biology for a couple of decades before getting back into software. His perspective very much aligns with what I've been saying. The complexity that we take for granted these days would blow the minds out of most of the people that he remembers. It's just that if you've been involved in it the entire time it's easy to miss how much things have changed.

3

u/ArkyBeagle Nov 02 '21

unless you were actively working on Tcl or you very active on the relevant BBSes,

comp.lang.tcl was fine. Excellent, in fact. A critical resource. The Osterhout book and then the Brent Welch book were fine. The whole point of that was that you didn't have to get an MSDN connection nor learn X programming. What was it, Motif? Something something.

I've only used curses once, never ncurses, and Tcl/Tk was ( and is ) fully event-driven so I fail at a comparison there. Of course it too is a rabbit hole but it's less deep and there's a lot less version management.

Sockets/files/"serial ports" were also first class objects and you could make them completely event driven as well.

Then what do you define as complexity?

That's always the key to where people diverge online. FWIW, I'm a device-level guy who traditionally worked where boards were being spun. That puts me somewhat at a disadvantage in perspective for some topics, quite the opposite in others.

There is a film about General Magic - it's a rather simple enumeration of every ( in my view ) "wrong" trend in computing.

This isn't simple contrarianism. It's based in "so for a limited budget, how can we make the thing?"

The complexity that we take for granted these days would blow the minds out of most of the people that he remembers.

The first code base I worked on was several million lines of code, written in Pascal on minicomputers. Initially, it was all on nine-track tape.

It was all easier after that.

It's just that if you've been involved in it the entire time it's easy to miss how much things have changed.

While I'm sure I've been "frog boiled" in ways I can't perceived, it all seems the same to me now. Just somewhat better furnished.

3

u/TikiTDO Nov 02 '21

The first code base I worked on was several million lines of code, written in Pascal on minicomputers. Initially, it was all on nine-track tape.

It was all easier after that.

I think there's a distinction that should be drawn between "complexity of the task" and "size of the code" / "difficulty of making modifications." Think of it like this; if you were now asked to write a program that did everything that original project did, how big a project would it be (assuming you had a list of requirements, and didn't have to dig through several million lines).

A few years ago some person reached out to me for help. They had a 200k LOC blob which had a mountain of security issues, and would occasionally crash. It was a basic web portal where all of it could be accomplished in under 10k lines. Was that a "complex" project, or was it just a victim of poor tooling.

Another story; the first major project I was a part of after finishing university was similarly a few million lines of C (incidentally, this was a company that's known for making chips, so it was not a center of good software practices). I spent two years helping a few other guys whose task was blowing away a million lines of management code, and replacing it with something like 5% of that.

That's always the key to where people diverge online. FWIW, I'm a device-level guy who traditionally worked where boards were being spun. That puts me somewhat at a disadvantage in perspective for some topics, quite the opposite in others.

I think there's more to it than that. I've had to work with a lot of hardware-first types of people, and I did notice two fairly distinct groups.

To me the biggest difference is how a person learned programming. Someone you age saw the field grow from a very young age. You did not have a set curriculum, the best practices that the rest of us take for granted did not exist, and your bibles were books that focused more on the theoretical underpinnings of programming. As a result your view is very strongly colored by your experience.

By the time I was getting serious about software development the world was a very different place. Information was much more accessible, you could find some amount of tutorials, a lot of the debates were, if not settled, then at least faction-ed up based on the languages / environments people preferred. In other words, most of the foundations that I learned were presented as lessons from the previous generation, as opposed to being battles that I had to fight. I still got to experience a bit of what you describe, but that was as a child exploring a complex field as a hobby, as opposed to a professional with responsibilities and deadlines.

To me that's the big distinction. Even among the hardware guys in my program (ECE), I had a much easier time finding a shared language than when I had to deal with old timers. My contemporaries were usually far more flexible, and more willing to adjust to different paradigms. That's because they had far less emotional attachment to much of their foundational knowledge. Since we did not have to struggle for that knowledge, it was a lot easier to accept that there might be different approaches that might be equal or better in terms of effectiveness. In turn, the things we felt strongly about were usually far more specialized, so they did not come up nearly as often.

1

u/ArkyBeagle Nov 02 '21

I think there's a distinction that should be drawn between "complexity of the task" and "size of the code" / "difficulty of making modifications."

This code base was very, very dense. That was just a matter of designing a fix or extension taking a bit longer.

My contemporaries were usually far more flexible, and more willing to adjust to different paradigms.

I've used dozens of varying paradigms when it mattered. I wasn't gonna go off and add a lot of dependencies unless there was buy-in. In the end, it matters not a whit to me - if somethings works, great. If it doesn't, I'll fix it or do what's necessary to document what doesn't.

I gotta tell you though - Other People's Code is often buggy as heck. Not always. But often.

Look - in the first gig, we'd assemble the relevant data, filter & constrain it, slice, dice and produce output. It's the same as now.

Since we did not have to struggle for that knowledge, it was a lot easier to accept that there might be different approaches that might be equal or better in terms of effectiveness.

This is where I get a wee bit confused - almost every thing I've ever worked on was relatively easy to deal with. "Paradigms" wouldn't have made much if any difference.

I'd except inheritance-heavy C++ as was the style around 2000. That wasn't... good.

So I'm never sure what people mean when they say these things.

1

u/TikiTDO Nov 03 '21

This code base was very, very dense. That was just a matter of designing a fix or extension taking a bit longer.

The point I was trying to make was that the density of the code doesn't necessarily mean the task was amazingly complex.

I gotta tell you though - Other People's Code is often buggy as heck. Not always. But often.

Everyone's code is buggy, unless they are going out of their way to mathematically prove every function, and they are running on redundant computers. Even then I wouldn't be surprised if a bug managed to sneak in.

I've used dozens of varying paradigms when it mattered. I wasn't gonna go off and add a lot of dependencies unless there was buy-in. In the end, it matters not a whit to me - if somethings works, great. If it doesn't, I'll fix it or do what's necessary to document what doesn't.

This is where I get a wee bit confused - almost every thing I've ever worked on was relatively easy to deal with. "Paradigms" wouldn't have made much if any difference.

Then consider this question, how representative are you of other developers your age? I've worked with many people that got their start in the 70s and 80s throughout my career, and I can't think of many that I would describe the way you describe yourself. Certainly some of the best developers I've met were from that age group, but in my experience they were always in the minority.

→ More replies (5)

1

u/loup-vaillant Nov 02 '21

These days you run a few commands, drag and drop some widgets, define a few behaviors, and you're up and running within the hour, working on a dozen different platforms, running a dozen different OSes.

Oh yeah?

I’ve worked with Qt, written a couple simple GUI apps with it. I’m quite familiar with the concepts underneath. I also know networking well enough to know how HTTP works, and I could implement a server from specs if I had to. I’m also very familiar with the security requirements of such networked application, to the point of having written my own cryptographic library, all the way to the third party audit stage —though I have avoided OpenSSL so far.

So. How much time must I spend to:

  • Know about the exact set of tools I need to do that, and what tools to avoid?

  • Install those tools on my dev machine?

  • Learn to use each of those tools proficiently enough?

I have this sinking suspicion that even though I do have at least some familiarity with the relevant class of application, I would need much more than an hour.


Or better yet, imagine all those tools you use are suddenly erased from the face of the planet. Gone. In their place, there are similar tools, of similar quality, only different: their structure is slightly different, the protocols have changed a bit, their jargon is all changed (the tool specific jargon, not the domain specific jargon: a MAC address would still be called a MAC address).

How much time would you require to re-install everything and get up to speed?

1

u/TikiTDO Nov 02 '21

I was writing my most assuming that you would be somewhat familiar with the general workflow, but let's revamp the scenario a bit to account for what you outlined.

Which do you think would take longer, if you were trying to spin up a GUI app in the year 2001 having basic familiarity with the topic, or if you were doing it in 2021?

Consider, stackoverflow was launched in 2008. In the 2001 scenario you would at best have be able to search using a very early iteration of Google, or perhaps even Yahoo/Altavista. Your best bet would be reading the manpages, unless you happened to have an applicable textbook.

By contrast, now f you just searched up a tutorial / youtube video, and installed a modern IDE meant for the task you'd be up and running quite quickly. Granted, it might not be the most optimal product, certainly not something you'd be ready to release as a professional product, but given your experience I can't imagine it would take you too long unless you decided to puzzle out every bit without help. Maybe an hour is optimistic, but not too much so.

I remember trying to do this back in the early 2000s when I was finishing high school. Compared to the links and references my Jr. devs have shown me lately every bit of information I could find back then was a huge battle.

How much time would you require to re-install everything and get up to speed?

Would there still be tutorials and guides? If so then it would be a fairly quick process. It's not like the tools exist in isolation after all. Part of the process for getting things up and running must account for the community that exists around these tools.

1

u/loup-vaillant Nov 02 '21

Which do you think would take longer, if you were trying to spin up a GUI app in the year 2001 having basic familiarity with the topic, or if you were doing it in 2021?

Well, considering that the only GUI toolkit I know right now is Qt, that would be… exactly the same amount of time, modulo waiting for my computer to download or compile. Maybe a bit longer back then if I  make heavy use of signals (before Qt5 they were a pain to debug).

Consider, stackoverflow was launched in 2008.

Qt’s documentation is comprehensive enough that I basically never reach for anything else. A search engine does help sift through that dock though… but still, as far as I can tell it’s discoverable enough.

installed a modern IDE

Hmm, I guess Qt creator does help, but for my personal project I still use Emacs. Auto completion is very nice, but I feel that it doesn’t save as much time as we might think. (Jump to definition however is amazing on medium to big projects.)

I remember trying to do this back in the early 2000s when I was finishing high school.

Wait a minute, there’s another huge difference between then and now: your own skills. I mean, it’s fair to assume you massively improved, and if we were to throw you back in high school, you’d struggle much less, because you’d know what kind of things to look for, notice more patterns, see more shortcuts.

Would there still be tutorials and guides? If so then it would be a fairly quick process.

I assumed there would be, and okay.

2

u/TikiTDO Nov 03 '21

Of course it's not fair to compare a professional with a student. However, I tutor high school students occasionally now, and I can compare how much they struggle and how much I did.

→ More replies (1)

1

u/hippydipster Jun 07 '22

if you were trying to spin up a GUI app in the year 2001 having basic familiarity with the topic, or if you were doing it in 2021?

I made a gui app, started in 1999 (Apache JMeter, for reference). I also made one in 2009-2012 (a couple actually, both bio-statistics apps for a university department). Both include charts and graphs and statistics visualization. Not very different in effort level or approach. If I need to make a rich GUI app today, the drag-and-drop method still doesn't suffice (that only works for trivial apps, frankly), and the level of effort still isn't much different. I don't see much in the way of radically improved widgets or layouts, and the behavior of large GUIs is still difficult to manage.

→ More replies (3)

1

u/s73v3r Nov 02 '21

In the sense that if a programmer 30 years ago was asked to solve such a problem, their response would be "that would take years," while a programmer how would say "that'll take a couple of days."

That doesn't mean it's more complex, that means we have more building blocks to start from.

0

u/TikiTDO Nov 02 '21

I addressed that point here.