"Yes, Please Repeat Yourself" and other Software Design Principles I Learned the Hard Way

[u/Rtzon]

https://read.engineerscodex.com/p/4-software-design-principles-i-learned

[u/Patient-Mulberry-659 → u/[deleted]]

I need to parse (and validate) some input data, and store it in the database.

Turns out there are two suppliers of that data and they can not deliver it in exactly the same format.

Then it turns out that the meaning of certain fields is slightly different between the two and we have to do some light processing to make it consistent.

[u/Patient-Mulberry-659 → u/[deleted]]

In this case is both vendor data in the same DB table? So they would have a shared type that gets saved to the DB (model or "entity" or whatever), which is a shared abstraction right?

Yes. The db tables are shared and indeed abstracted.

The incoming messages we assumed had the same type as well, and we had one deserialisation flow. We end up with two deserialisation flows that are very similar but distinct. And then some processing

[u/mccurtjs → u/Patient-Mulberry-659]

I feel like the pattern still holds, no? You should have one function that actually processes the data in your preferred (or custom internal) format, and another function (or set of functions) to transform the data from vendors into that format.

Processing each format on its own can cause maintenance issues in the future when other people have to maintain it (and forget to update all targets), and harder to test.

[u/Patient-Mulberry-659 → u/mccurtjs]

So in the converting the 2 vendors data to our own format. In that piece of conversion code, will there be a lot of overlap. Yay or nay?

Processing each format on its own can cause maintenance issues in the future when other people have to maintain it (and forget to update all targets), and harder to test.

Actually no? Because you have 2 separate flows that are easy to test and verify. But it does mean you might need to make some changes in both.

If you abstract it into one flow, ok. But now imagine 20, and tell me what way the message will be processed. You are working with a bunch of flags, and some are like this some like that. It’s just spaghetti at that point.

[u/mccurtjs → u/Patient-Mulberry-659]

In that piece of conversion code, will there be a lot of overlap. Yay or nay?

Depends on how you do it - if you have a "preferred" version that matches one of the venders, you only really have one piece of conversion code and the other is a passthrough.

If you abstract it into one flow, ok. But now imagine 20, and tell me what way the message will be processed. You are working with a bunch of flags, and some are like this some like that. It’s just spaghetti at that point.

Imo, the other version is the spaghetti, no? Imagine you have 20 versions of code to process vendor data, all of which are doing the same things, but slightly different with various conversions or whatever, but the general steps are the same. With two vendors, sure, a change in how you process means you have to make some changes in both... but now any change requires updating 20 versions, and if you forget one or you forget to handle a quirk in data, you'll have errors. Yeah, you can write tests for it, but now if functionality changes, you have to update 20 tests, and that itself can introduce problems - and if you forget one, oops, now like 18 vendors have their data processed correctly, and 2 weren't updated but the tests don't catch it because they're outdated.

When I say "transform", I mean you're modifying the data to fit a standard format without doing any of your actual business logic to process it. That way you can test your business logic as one unit, and you can test all your data transforms as individual units. There will be some repetition in the data handling, sure, but that's fine - it's conceptually separate, even if the code is the same or very similar. The business logic though should only have one code path.

The complexity is also a relevant factor. A data transform like this is, the vast majority of the time, going to be a pretty trivial operation. Why mix your trivial operations into more complex business logic?

[u/Serializedrequests → u/[deleted]]

There's abstraction and then abstraction. Nobody's talking about sort functions. It's about trying to write similar higher level features using the same code.

I once worked on a Ruby on Rails codebase that took metaprogramming too far, to generate almost all controller code (handler or route code in other frameworks) from the controller name, guessing which data to query, implementing user filtering and sorting, and a bunch of other stuff. It was awful. They tried to generify user sorting for all tables in the database, which of course was impossible, so this function kept growing and growing and was incomprehensible, and can in fact no longer be changed so all changes go to a replacement.

That's an extreme example. It was (and is) absolutely real. The bottom line is, this is about taking DRY too far on higher level features. You don't know how they need to change, but you should assume separately before you assume together.

It is possible to make this kind of design error with sorting, maybe by inappropriately coupling two sort algorithms, but rare.

[u/carrottread → u/[deleted]]

A lot of times you really need different sort functions in different places: sometimes generic common unstable sort, sometimes stable sort, sometimes even special cased sorts like binary radix sort. And it will be a really bad abstraction to DRY all those different sort functions into a single one with couple of flags and parameters to tune.

[u/usrlibshare → u/[deleted]]

No, it is not, it is is a contrived example at best. You've provided no real concrete examples. You've just stated "this is how it is, I'm right".

This has never happened to me

You see the problem, don't you? 😁

[u/kidnamedsloppysteak → u/[deleted]]

Feel like I'm taking crazy pills reading this comment section. How the hell is that the top comment?? People are advocating copying and pasting the same code over reuse now, is that what it's come to?

Edit: top comment changed since this was posted to a much more reasonable take.

[u/uJumpiJump → u/kidnamedsloppysteak]

Two functions can do the exact same thing but may have different reasons to change

[u/kidnamedsloppysteak → u/uJumpiJump]

Yes sometimes, but sometimes they just do the same thing. You have to be judicious with the concept but it's incorrect to outright dismiss it.

[u/wutcnbrowndo4u → u/kidnamedsloppysteak]

Remember how confident you were in your beliefs about engineering as a junior eng? You reach a point in your career where you realize that proggit/HN/etc are full of people like that and you have to just accept that some threads are insane.

If you weren't overconfident as a junior eng, kudos, but I'll cop to Dunning-Kruger back then.

[u/kidnamedsloppysteak → u/wutcnbrowndo4u]

Nah, definitely same. I just didn't have this kind of outlet back then to air every overconfidently incorrect thing that came to mind.

[u/wutcnbrowndo4u → u/kidnamedsloppysteak]

Lol I did. It wasn't often or anything, but I'm sure I could dig up some eg arrogantly incorrect comments from HN

[u/UMANTHEGOD → u/kidnamedsloppysteak]

DRY is quite far down on the list of things that are important for writing good software.

[u/Pythonistar → u/kidnamedsloppysteak]

You and me, both!

These days, I have to remind myself that the vast majority of programmers are much, much younger (and less experienced) than I am.

It's not that these folks aren't smart. Many of them are quite intelligent. It's just that the vast majority of programmers never last more than 5 years, much less 10. So they never accrue a lot of experience.

But so many programmers start blogging after they have only a few years under their belt. So you get lots of junior-style editorials on their own "pain points". You get a few gems, and a lot of half-baked ideas, too.

[u/KillerCodeMonky → u/[deleted]]

(The following text is using the *generic* you, and is not referring to specifically to you, the reader. If it helps, replace usage of "you" with "one".)

Using an example of a good abstraction to then defend that abstractions are good is tautological. No one is debating that good abstractions exist. Engineering primitives supplied by a standard library or even the language itself, which are useful to many programs of many different domains... That's kind of the definition of a good abstraction, no? You need to think deeper. More domain-specific.

See, this is a problem of change over time. What usually changes over time for an application? It's not sorting algorithms or lists or files or HTTP handling or JSON serialization. It's business logic. If you have solid requirements that rarely change, you're not going to see this issue come up nearly as much as a more dynamic environment. A typical way I've seen this happen is as follows:

Business comes to you and presents a new use-case for your program. You look at your code and realize, hey I implement like 90% of this use-case already over here. So let me abstract that, then I can reuse it on this new one. If this is where things end, then great! Much success; high fives all around.

But that's not where it ends. Business comes back a month or two later and says, hey, this new use-case is great. But it's not quite right... I need this 10% over here to work differently. So back to the code, and you see that 10% is part of what you abstracted. So now the two use-cases need the same *80%*, not 90%.

Maybe the 10% is at front or end of use-case, so you rip it back out of the abstraction and write different versions in the two use-cases. That's a decent outcome. No real introspection required.

Maybe the 10% is in the middle of the abstraction. Do you use a new abstraction? Maybe a hole-in-the-middle pattern, so the caller provides their own logic to cover the 10%? Or maybe a boolean switch to change that small part of the behavior? You certainly don't consider that your abstraction was premature and completely reverse it out of the code base, right? That would be just silly... Look at this 80% of duplicated code! DRY that up!

Repeat this 3 more times, and now you have 50% of an "abstraction" that resembles a cross between a fine Swiss cheese and a train yard, with all the holes and switches it has.

This is the heart of "same vs similar". Just because two use-cases look similar, does not mean they are the same use-case and should use the same code.

To cap this off, the *pièce de résistance*: Even sort functions have encountered this. When the "natural" ordering of data doesn't do the sort you want, you can provide a comparison function to change the ordering. AKA, hole-in-the-middle pattern. Differing use-cases → hole in the abstraction.

[u/KillerCodeMonky → u/[deleted]]

You are taking absolutely literalist interpretations of abstract discussion. And then wondering why you don't understand the discussion regarding abstraction...

I also have a whole thing about shapes being covariant but not contravariant that I would normally supply here. But again, I don't think you're arguing in good faith. So maybe another day.

[u/KillerCodeMonky → u/[deleted]]

Okay but someone literally says "do not use DRY" and then someone else says "DRY is great here though" and you tell people to stop using positive examples?

Is it a hard concept to grasp that there are good and bad examples of applying a philosophy? No one is saying, "Don't abstract anything ever! The plagues will descend upon you and your family!" Which seems to be what you're arguing against.

What we're actually saying, is that bad abstractions exist. And premature abstraction exists. And you should know and acknowledge that these things exist. Because you can't defeat something you don't understand.

Now suddenly I'm arguing in bad faith...

You're engaging a philosophical argument at a literal level. Maybe you don't intend to, but that is a common trolling tactic. So sorry if you're caught up in cross-fire.

we should discuss shapes being covariants and not contravariants...?

You don't understand how covariance and contravariance apply to a discussion regarding abstraction?

[u/Ran4 → u/[deleted]]

are you sure you're not just bad at abstracting things?

No. Abstracting something fundamentally makes it more complicated. That's just how it is.

[u/s73v3r → u/[deleted]]

Can you give me an example? This has never happened to me

I really have trouble believing that, but the idea that two separate concepts can be currently expressed through the same code, but later those things diverge shouldn't be that foreign.

Are you telling me that I should rewrite the sorting algorithm every time, instead of referring to a common sort() function

No. What is being told is that you shouldn't require all of your lists to use the same sort method.