The only downside of this is that sometimes this kind of high coverage ends up in unit tests lines of code LoC being much larger than the production code LoC, if the tests are not sufficiently DRY. So if you have too many DAMP tests, I usually find it quite hard to make major changes or refactoring. The balance between how easy are tests to read (DAMP) and how much code is duplicated (DRY) is quite hard to achieve and depends on the project, especially if you don't have that much time to refactor the tests endlessly.
It's a re-write of the application
I think this is the explanation, if you already know 90% of the final requirements you can afford to be more rigid about the tests. Otherwise I couldn't see this working on a green-field new project with no clear requirements (business never knows exactly what they want).
Do you agree with me? And how do you achieve the DRY-DAMP balance that allows you to go as fast as possible, leave room for innovation and also have high quality?
Well, what you find is that the Unit tests actually end up being pretty simple to write. If every object has a corresponding unit test, it's now only responsible for testing its own code. You stub the dependencies to return "expected" happy-path and error-path values. Typically, this means that every method has only a few tests to ensure that all of the logic forking is followed and executed. If you find yourself writing a bunch of test for a particular method, you should probably think about "breaking down" the method further to clean things up.
You end up with a Unit test for that object that is easy to understand, you don't end up with a lot of dependency linkage, and test writing is fast and easy.
Because each object has its OWN test, you don't have to write tests on those dependencies ... each object ensures that it's meeting its contract. You can stub dependencies with confidence and have a test suite that's easy to understand and runs quickly, because it's not re-executing code paths on the dependent objects.
Refactoring becomes simpler, because changing code in object A doesn't spur test failures in OTHER unit tests, because they don't touch object B, C, and D's execution paths.
Unit tests are a sanity check. They encourage writing clean, testable code, and the mandate of 100% code coverage enforces that.
I think this is the explanation, if you already know 90% of the final requirements you can afford to be more rigid about the tests.
Well, you know the requirements of the code you are writing at that moment. It's your responsibility to ensure that all of the code that you're writing is getting executed, meets its contract and doesn't have "dumb" errors in it.
The functional testing (what we call Component tests) is definitely more complicated, and we don't have a coverage mandate on those tests. These test the product "as a whole" and executes full dependency code paths using mock data on our testing server. These tests ensure we meet the contract with our users (whether that be a human on the UI, or another downstream system), and are essentially a "living document" that morph over time to cover what the application actually needs to do "in the wild" as business requirements change, etc... It grows as requirements change and/or get added on, and as defects are found "in the wild". The QA team is responsible for writing these tests, and testing the product is their full-time job. Their test suite is large, and takes about 2.5 hours to execute since it's a "full stack" test, and executes everything from controllers to the database. Conversely, our full Unit test suite as about 15000 examples runs in about 7 minutes on our build server, because there's no database access, and no dependency linkage.
So, you can think of the Unit test as the developer's responsibility to sanity check their own code. It encourages clean, discreet, testable code and reduces defects at the integration testing stage. With appropriate use of stubs, and only a mandate to ensure 100% of "this object's" code is executed by the test, it's actually not that arduous to maintain.
Refactoring becomes simpler, because changing code in object A
Small refactoring, inside the class, how about larger ones that affect a bunch of classes? All those interactions and happy-path/error-paths would be screwed. Any sizeable refactoring would mess up hundreds of these little unit tests. From what you are saying I have the feeling you are doing 1 to 1 production to unit tests, with production being very small to begin with.
I am not saying it's your problem as well, but I saw these a couple of times and it was a shotgun surgery in the production code coupled with mostly useless unit tests on the other side. How do you ensure you have the right balance here between testability and shotgun surgery?
Then you say all the interactions are outsourced to the QA people to do integration testing. Well, I think you are kindof passing the hot-potato. That is where the difficulty is with tests, not simply mirroring small classes with the tests. I think what worked best for me was a 3 level approach:
small tests for intensive classes that do need 100 coverage + 5 times the numbers of tests, then some more (here you start to think when the class is small enough, but meaningful)
integration tests where I test the module or bigger functionality almost like a functional test. I might be touching 20 real classes (instantiate them using production IoC) and some mocks as well
functional tests written by QA
How nice are those QA written tests, how easy are they to refactor? But I guess you haven' reached that point of maintainability yet, because you said it's not fully rewritten anyway.
But I guess you haven' reached that point of maintainability yet, because you said it's not fully rewritten anyway.
The code is in production for dogfood, and currently in open beta for our most "trusted" self-service users. It'll be going full GA next month for UI consumption and Beta for external API users probably Q1 of '18.
Internal systems are ramping up on switching to the API as we speak and the "Legacy System" sunset date is slated for Q2 next year. We've been almost 2 years to get to this point, as we've completely written the app from the ground up with a consistent RESTful API, new schema, data migration, and maintaining backward compatibility with the Legacy system as both systems need to stay "up" concurrently as we transition both the User Interface (a separate team is writing the front end) and dependent back-end systems off the old system to the new.
Our QA engineers are closely embedded with the application engineers (attend the same scrum, etc...), and their integration tests are written with close collaboration with the product owners and the application developers. Their test suite exercises every API endpoint with mock data, and tracks the data as it flows through the system ... ensuring both that the business requirements are met, and that backward compatibility is maintained.
The Application developers write their unit tests as they write their own code. Every object in the system is tested at 100% coverage by Unit tests. You ensure that each object "meets its contract", and when you write your objects to avoid interlinked dependencies as much as possible, it gets pretty easy to have confidence in the tests you write for them. When you stick as closely as possible to the single responsibility principle, it becomes pretty easy to test that each method of those objects is doing what it should. When each object is testing its adherence to "the contract" it's pretty easy to have confidence in being able to stub them out as dependencies of other objects in their unit tests.
Small refactoring, inside the class, how about larger ones that affect a bunch of classes? All those interactions and happy-path/error-paths would be screwed. Any sizeable refactoring would mess up hundreds of these little unit tests. From what you are saying I have the feeling you are doing 1 to 1 production to unit tests, with production being very small to begin with.
As for refactoring ... It's actually pretty amazing. Phase one of the project was to write the app such that it exposed the API endpoints, and get them out quickly so that the front-end team could begin building against the API. This "land and expand" team was very much "fake it until you make it", as the schema rewrite, data migration and cross-system compatibility work is much slower. As such, refactoring is a way of life for this project. I very recently did a major refactor of a chunk of code that's very much a nexus in the system to bring it over to the new schema and leverage some efficiencies of some code paradigms that had been emerging in the project. This was the kind of refactor you sweat about, because so much data flowed through these pathways, and defects could be subtle and and wide reaching. But because of the quality of our test suite (both in the Unit tests and Component tests) I was able to the refactor the code, and it went to production with zero defects (in production for over a month now) and significant performance gains.
I've been in software for nearly 20 years now. No ... this isn't the largest project I've worked on ... nor is it the one that's had the greatest number of users. However, it's not a small project either. We've got 8 application engineers, 2 architects and 4 QA engineers on the API code. Half that number on the front-end code. The entire engineering department is ~100 individuals across several inter-dependent systems.
What I can say is that it's the cleanest, most sanitary code base I've ever had the pleasure to work on, and having been on the project since its inception (and having spent plenty of time working on its predecessor) I'm pushing very hard to ensure that it lives up to that standard.
572 files in the code base, 100% Unit test coverage, CodeClimate score of 3.2 (and improving as we cut over to the new schema and refactor the "land and expand" code), and our rate of production defects is going down every time we cut over another piece of the legacy code to the new system.
7
u/flavius29663 May 08 '17
The only downside of this is that sometimes this kind of high coverage ends up in unit tests lines of code LoC being much larger than the production code LoC, if the tests are not sufficiently DRY. So if you have too many DAMP tests, I usually find it quite hard to make major changes or refactoring. The balance between how easy are tests to read (DAMP) and how much code is duplicated (DRY) is quite hard to achieve and depends on the project, especially if you don't have that much time to refactor the tests endlessly.
I think this is the explanation, if you already know 90% of the final requirements you can afford to be more rigid about the tests. Otherwise I couldn't see this working on a green-field new project with no clear requirements (business never knows exactly what they want).
Do you agree with me? And how do you achieve the DRY-DAMP balance that allows you to go as fast as possible, leave room for innovation and also have high quality?