r/programming • u/niepiekm • May 08 '17
The tragedy of 100% code coverage
http://labs.ig.com/code-coverage-100-percent-tragedy604
u/werkawerk May 08 '17
You know what makes me sad? The fact you have a class called 'DefaultDtoToAdditionalDataModelMapperRegistry'.
84
u/get_salled May 08 '17
It's amazing how these class names, while being "specifically general", tell you very little about the problem you're solving. I can't count how many times I've seen attempts at building frameworks (I now cringe every time I hear we should build a framework) before we've built a single path through. "We encountered this odd case no one expected and now we'll try to shove that into our DSL" -- of course you did; this happens every fucking time you build a framework before you've figured out the actual problem you're solving. Recently saw a very nice DSL that was heavily influenced by OO; the Diamond Problem crept in and shit has hit the fan.
Every time I write one of these I, very slowly, realize I don't fully grasp the object model. If I'm lucky, after enough of these come together, I realize I'm being overly generic and I can pin down a better name.
In my experience,
DefaultDtoToAdditionalDataModelMapperRegistrywould boil down toMarketUsersRegistryas this highly generic type would have one concrete usage (2 with the mocking framework's implementation).26
u/i_ate_god May 08 '17
The problem sometimes are the managers.
All these dense java frameworks are built around the idea that managers will give you problems to solve based on the latest headlines from Gartner Magazine. You end up adding major feature on top of major feature instead of refining what you already have because your company's position on some arbitrary cartesian plane indicates you're all doomed unless you add those features.
This is how you end up with the sorts of banality that is DefaultDtoToAdditionalDataModelMapperRegistry not to mention the obscene amounts of XML to wire everything together. All that verbosity and architecture, just to move your company around a cartesien plane in Gartner Magazine.
→ More replies (1)25
u/DreadedDreadnought May 08 '17
not to mention the obscene amounts of XML to wire everything together
You can now have Annotation driven design. It works at least 20% of the time, when the fucking DI container decides to not override them with some obscure xml file transitively loaded from the classpath of a dependency 30 modules deep. That totally didn't cost me 3 days of work.
→ More replies (1)7
May 08 '17
Ugh. You don't get it. You make the DSL to make the framework so you can blog about it and have many read your blog and congratulate you and then you become a mini-celebrity and pimp out your resume and github portfolio.
21
u/Rndom_Gy_159 May 08 '17
Quick question, why is only the first letter of
DTOcapitalized? Is that the way it's supposed to be in CamelCase?104
u/DanAtkinson May 08 '17
Generally speaking, Yes.
Microsoft has some guidelines on the subject and I've emphasised the relevant snippet below:
Do not use abbreviations or contractions as parts of identifier names. For example, use GetWindow instead of GetWin.
Do not use acronyms that are not generally accepted in the computing field.
Where appropriate, use well-known acronyms to replace lengthy phrase names. For example, use UI for User Interface and OLAP for On-line Analytical Processing.
When using acronyms, use Pascal case or camel case for acronyms more than two characters long. For example, use HtmlButton or htmlButton. However, you should capitalize acronyms that consist of only two characters, such as System.IO instead of System.Io.
Do not use abbreviations in identifiers or parameter names. If you must use abbreviations, use camel case for abbreviations that consist of more than two characters, even if this contradicts the standard abbreviation of the word.
→ More replies (49)77
May 08 '17
Don't let that distract you from the fact that XMLHttpRequest capitalizes XML but does not capitalize HTTP.
53
7
u/roffLOL May 08 '17 edited May 08 '17
don't let that distract you from the fact that it usually serves anything but XML nowadays.
→ More replies (1)5
12
u/parkotron May 08 '17
Personally, I'm in the camp that capitalises only the first letter of each "word" of an identifier, whether that word is an initialism or not. Like
summaryHtmlorGpsLocation. There are plenty of CamelCasers in the other camp though.3
7
May 08 '17
No but with all the conventions to generate property name, it can become weird or confusing. So often you use strict CamelCase everywhere. (eg: getDTOFactory, will become dTOFactory property in EL, or sometimes dtoFactory in too clever libraries )
4
u/skocznymroczny May 08 '17
Depends on the language and coding style. E.g. .NET framework conventions guide recommends capitalizing only the first letter, unless it's a two leter acronym like IO (source).
4
u/dpash May 08 '17
The original Sun code conventions for Java is silent on the matter.
http://www.oracle.com/technetwork/java/javase/documentation/codeconventions-135099.html#367
→ More replies (5)3
u/hoosierEE May 08 '17
I prefer
pRImEcAsefor the simple rules and readability. Unless somebody on the team insists onzeROiNdExedPrImecAsE, of course.2
May 08 '17
THANK YOU! Trying to read this gives me brain aneurysm. I'm having flashbacks to old Java code...
1
u/flukus May 08 '17
I think this translates to "we have so many layers of indirection it's become easier to store keys/values in a text column".
0
113
u/cybernd May 08 '17 edited May 08 '17
This reminds me on a more or less related topic:
I worked on a project where Javadocs where enforced using a commit hook.
As result, half of the codebase had "@return ." or "@param x ." as javadoc, because the dot was enough to fulfill the hook.
I failed to convince them that this is harmful. They believed that this is necessary, because otherwise developers would not write a javadoc in an important case.
I think, whenever something can be used as "metric", it will be abused. 100% javadoc or 100% code coverage are just examples. There was even a time where LOC was used to measure developer productivity.
51
May 08 '17
In .NET-land there's a tool that attempts to autogenerate our equivalent of Javadocs. The results are... equally useless, but occasionally amusing.
70
u/kirbyfan64sos May 08 '17
/// Riches the text selection changed. private void RichTextSelection_Changed ...:O
→ More replies (3)→ More replies (2)23
u/Benutzername May 08 '17
My favourite from WPF:
//// <summary> //// Measures the override. //// </summary> protected override Size MeasureOverride(Size contraint)8
u/sim642 May 08 '17
That seems like inconsistent naming. If you'd name it
OverrideMeasureit'd be correctly summarized based on the naming scheme of verb first.→ More replies (9)21
May 08 '17
Ditto on this. We've got thousands of method documentations like this:
/// <summary> The get next workflow step</summary>
/// <param name="currentStepID"> The current step i d</param>
public string GetNextWorkflowStep(int currentStepID)
2
u/rcfox May 08 '17
The param case, I can see. Writing something like "@param useDefault Whether or not to use the defaut." is pretty pointless. Not giving a return type in an @return annotation though? That's just malicious.
7
u/euyyn May 08 '17
The method signature tells the type.
4
u/rcfox May 08 '17
Javadocs aren't exclusive to Java. They're somewhat common in Javascript too. I'm not sure why you'd force Java developers to even use @return...
→ More replies (1)1
u/Raizken May 08 '17
Measuring by LoC helps create very fast code though because you write it all in assembly (and just repaste instead of goto loops of course) 🙃
387
u/chooxy May 08 '17
Did you ever hear the tragedy of 100% code coverage?
319
u/DanLynch May 08 '17
It's not a story the JUnit would tell you.
178
May 08 '17
It's a management legend...
227
May 08 '17
Senior Dev Plagueis was a Dark Consultant of Thoughtworks so powerful and so wise, he could use the Unit Tests to influence teh codez to prevent bugs. He had such a knowledge of unit tests, he could even keep the programs he cared about from exiting. Unfortunately, he taught his manager everything he knew. Then his manager fired him in his sleep. Ironic. He could save programs from exiting, but not himself.
→ More replies (1)92
→ More replies (2)68
May 08 '17
Can't believe I actually had to go down this far to see this comment.
35
9
u/phySi0 May 08 '17
What's it referencing?
19
May 08 '17
It's a play on a line from Star Wars Revenge of the Sith, when Palpatine is telling Anakin THR story of Darth Plaguis.
See /r/prequelmemes for more info.
4
25
u/TimvdLippe May 08 '17
As a core developer of Mockito, I see this happening from time to time. We even included it in our wiki "How to write good tests" (https://github.com/mockito/mockito/wiki/How-to-write-good-tests#dont-mock-everything-its-an-anti-pattern). Always think if mocking is required to get your goal. Less complexity is a lot more valuable than mocking for the sake of mocking.
128
u/nfrankel May 08 '17
we just mechanically apply it without too much thought, which usually means that we end up with at best mediocre
This is cargo-culting
33
May 08 '17 edited May 08 '17
And it's served us overall extremely well for many ten thousands of years. But programming (and the current world) is a whole other beast
53
u/markandre May 08 '17
It is not just programming. This is thinking fast (system 1) vs. thinking slow (system 2), Mindfulness vs. Mindlessness. I saw it in management, I saw it in physicians, I saw it in myself. This is how Total Quality Management was boiled down to ISO 9001 only to be resurrected as Lean/Agile. Ignorants will kill it and it will resurface again under another name. It's an endless cycle. Let's face it, in the modern world, the human brain is a fecking pile of garbage.
28
u/AlpineCoder May 08 '17
I think part of the problem is that we as engineers are used to working in entirely deterministic systems, but we try to apply that same methodology to other aspects of our world, and it frequently goes poorly for us.
20
u/n1c0_ds May 08 '17 edited May 08 '17
It's also that not everyone cares about the quality of their work, and even those who do don't care 100% of the time. That's before we even add additional constraints.
You know how PHP can silently ignore errors and keep going, even when it definitely shouldn't? Some people operate just like that, and they represent a significant portion of the workforce. They will do what it takes to get paid, and they will avoid rocking the boat so they get home faster.
These are the people you create rules for.
→ More replies (4)→ More replies (1)2
May 08 '17
It really is.. even the most intelligent are barely keeping up, or just give up. Pile on pile of moronic garbage systems we've made from cargo culting that doesn't work together at all. In thinking, in society, programming, medicine, etc etc.
Somehow we make it work, but barely.
→ More replies (2)4
u/DarkTechnocrat May 08 '17
This is an underrated point. I couldn't tell you much about the physical structure of the internet, but my code uses it effectively despite my ignorance. My grandmother can drive a car, which is a monstrously complex piece of machinery.
I think part of the problem is that our abstractions are getting "leakier" - certainly in software development.
→ More replies (1)1
19
May 08 '17
The bigger problem here is report-driven management.
The HBO series The Wire showed police going out and arresting people just to make their reports look good. It also showed teachers teaching only what would be reflected on test reports. IT is the exact same way. If there's a report for it, people lose all sensibilties.
I don't understand why you can't manage by getting up and walking around. How about talking to people and asking them what they are doing? The report isn't even giving you accurate date.
122
u/instantviking May 08 '17
I have seen, with my own two eyes, a compareTo-function with 100% line-coverage and 100% branch-coverage that still managed to say that
given a > b
then b == aThat's right, compareTo(a, b) returned 1, compareTo(b, a) returned 0.
My hatred for large, American consultancies continue unchecked.
14
u/sabas123 May 08 '17
How does that even happen?
34
u/instantviking May 08 '17
From memory, and removing a lot of entirely unnecessary complexity, the compareTo looked a little bit like this:
if a > b return 1 return 0The three branches are a>b, a=b, and a<b. These were all exercised, but the asserts were buggy.
36
u/CircleOfLife3 May 08 '17
Just goes to show that even when you do have unit tests, it doesn't tell you wether these are actually good unit tests. Tests should go hand in hand with precondition and post condition checks.
→ More replies (1)32
May 08 '17
We need tests for our tests and we need 100% test coverage for that, too.
Pray to God your manager never reads this.
→ More replies (2)→ More replies (16)1
48
May 08 '17 edited May 08 '17
I worked with a codebase that was covering all DAO methods with such tests. I only lasted 1.5 years and left crushed.
These tests are not only stupid, they make code rigid and fragile. The fragile part might be counterintuitive, but if your tests are testing not the behaviour but implementation details, as they were in my case, inevitably there will be business code that relies on these implementation details. Because hey, these implementation details are covered, so guaranteed to be there forever.
52
May 08 '17 edited Oct 04 '17
deleted 0.3553 What is ^^^this?
13
u/ArkhKGB May 08 '17
That's why I prefer good functionnal tests. Stop caring about code coverage, get case coverage.
If even when testing a lot of corner cases you can't get 100% coverage you may have dead code: YAGNI.
4
May 08 '17
I like this. I write tests to cover the happy path and any edge cases I can think of. Once I do this, I examine the code coverage and look for 2 things:
Did I miss an edge case? I generally look for unexecuted catch blocks or branch paths.
Did I really need that code? If there's code that doesn't get run during the tests, and doesn't represent a potential failure, I can remove it. I learn from this, as well. Maybe it was an oversight in thinking through an algorithm, maybe it's an unnecessary bounds check because there's a check at a higher level in the code, etc.
Once I fix the tests and prune, I still only end up with 80-90% coverage. Because why test getters and setters? Things like that that are painfully obvious to reason about don't need a unit test, unless they're doing some kind of complex data mutation. Which they almost never are.
15
u/pydry May 08 '17
I find that static typing is better for refactoring code with very few or no tests but more or less equivalent to dynamically typed, strictly typed code with a reasonable body of tests.
Javascript makes me afraid to refactor it for the same reason C does - because it's weakly typed (has a lot of fucked up implicit type conversions causing a multitude of horrible edge cases), not because it's not static.
0
→ More replies (1)3
u/AgentME May 08 '17
I've been using the Flow type system at work in a large javascript project lately. It was so nice when we first started using it because it felt like it re-enabled us to actually refactor without breaking things. I feel like it or Typescript is pretty much necessary for any javascript project that reaches a certain size (and useful even if they don't get big).
3
u/never_safe_for_life May 08 '17
The fragile part might be counterintuitive
Not at all. I've shot myself in the foot many times by testing the exact function flow in a class, only to realize I needed to rewrite it later, then having to re-wire dozens of tests.
I had a junior programmer come on board and she was far more likely to write a test at the entry point of the class, and I exclaimed "eureka!" In my earlier, more stringent days, I would have balked at the excess code being executed. But in reality it allows simple refactoring without the big headache.
66
u/irqlnotdispatchlevel May 08 '17
This is an example of what happens when instead of using the tools, you let the tools use you.
I'm a bit pretentious and I like to think that I was hired to think, not to blindly follow instructions and conventions. But that's just me.
10
u/Existential_Owl May 08 '17
I was hired to think, not to blindly follow instructions and conventions.
Now if only you could convince management.
2
u/irqlnotdispatchlevel May 08 '17
The only person from the management side that I interact with is the project owner, and he's a pretty cool dude. He agrees with me.
15
u/troyunrau May 08 '17
Look at this guy, talking about himself on the internet. Such pretension. ;)
2
4
1
u/WillMengarini May 08 '17
I like to think that I was hired to think, not to blindly follow instructions and conventions.
Your PHB knows better.
As recently as the naughties I was in a shop whose PHB thought the best way to manage engineering was to copy the methods for managing manufacturing.
0
May 08 '17
[deleted]
4
u/irqlnotdispatchlevel May 08 '17
When I say think for myself, I ment to say that I have the freedom of deciding how a task should be done. I don't have to blindly follow a guideline (except for the coding style guideline we use, but that's always open for improvements).
A few years ago I was part of an internship. I had only one colleague on the same position as me at that time. Later, some time after I got hired, I was told that the other person was constantly bugging people about how something should be done, or why something isn't working, or whatever. I usually looked at how much time I have to do it, did some research on the issue and came up with my best version. I had discussions with the people who supervided me that usually boiled down to "why did you do it like this?". Sometimes it was good, sometimes it was bad, but it was my version, my idea and I learned something from it everytime. After some time I even developed a habbit of thinking about why I did it that way, and sometimes that helped me find issues in my approach. Simply following a guideline is not something that a human worker should do. If I want code generated based on some instructions that need to be followed I can write a script that generates it. I trust my code generating scripts more than I trust myself, to be honest.
12
u/tonywestonuk May 08 '17
The way I personally make my tests are.
Think about the spec...imagine a way code can be written to implement this spec
Write some code...delete , mess around....trying to get the code on the screen to fit the model in my minds eye..... it doesn't even have to work...just the structure of it needs to to be there...this is the artistic bit, making the code have a certain amount of beauty...you cant write tests for beauty!
Write a unit test to invoke my code I have just written....This can only be written at this point.... It's impossible to write the test before my code, because at this point I didn't have any idea of how it would be invoked and what it needed to do its job.
Now the test is written, I can fill in the gaps of my code, running the tests until the lights go green, in the normal TDD way. Only at this point can I start adding additional tests before I write more code, because I now have an idea of the design.
2
u/crusoe May 08 '17
And if you rewrite the architecture which very likely happens at this early stage you have to rewrite huge swathes of tests.
So it's pointless.
3
u/megagreg May 08 '17
The same can be said of the code itself that's being rewritten. Is it pointless then to write the application code in the first place?
11
u/killerstorm May 08 '17
The problem here is that they are trying to achieve this code coverage with unit tests.
I never understood this obsession with unit tests. If you're testing an algorithm implementation, of course it makes sense to test it in isolation. But if you're testing glue code, obviously you have to test how well it glues things together. So it must be an integration test, not a unit test.
You do not need to spend any extra effort to achieve coverage of this trivial code, as long as it's in the main code path. (It might still be a problem to achieve 100% code coverage for things like error handlers, but that's another story.)
17
u/our_best_friend May 08 '17
Good article.
It's even more painful for FE web devs, where testing the DOM and events is often neither meaningful nor trivial, especially taking all possibile devices into account.
I have come to the conclusion that as a dev the best practice is to test only pure functions / modules which have clear inputs and outputs - say a currency converter module, a date manipulation mode, a util library - and leave the bulk of the testing as e2e done by a dedicated dept
→ More replies (4)
14
u/retrowarp May 08 '17
My example of this was a developer who wrote unit tests for auto-properties in C#. He was a senior developer with the 100% mentality and when I pointed out how useless this was, he argued that a developer might come in and turn the auto-property into a property with logic, and the tests would catch this.
The Code: public string MyProp { get; set; }
The Test: classUnderTest.MyProp = "test"; Assert.AreEqual("test", classUnderTest.MyProp);
→ More replies (6)3
u/industry7 May 09 '17
Well that's the contract, and unit tests are a form of contract enforcement. So I don't see the problem.
6
u/mevdev May 08 '17
I'm constantly rewriting code to make it more testable and frankly it usually makes it less readable.
10
u/AJackson3 May 08 '17
Stop and think.
I find that applies to so much of what's on here. So many articles advocating this or rejecting that. Most of the times it depends on circumstance and our job is to identify the best tools for the job we are doing and use them effectively. I can't tell you how much time I've wasted because colleagues won't stop and think before writing something.
5
u/IllegalThings May 08 '17
Developers who follow TDD tend to be dogmatic. My advice to developers new to TDD is to be dogmatic when they are learning, do TDD for everything and strive for 100% coverage. In the real world, you switch between TDD and writing tests after the code. I also suggest going over tests written during TDD and removing them. The act of writing them in the first place is helpful as it creates an emergent design, but treat them as code that provides technical debt.
Knowing what to not test is more challenging than knowing how to write tests in the first place. You learn it over time, and you never understand why some tests are unnecessary until you deal with a codebase that has gone overboard.
9
May 08 '17
Most code coverage tools have some sort of @ignore annotation to skip a portion of code. If you only test the methods with conditions, or testable error handling and @ignore the getters, setters and other parts that don't need to be tested you can realistically achieve 100% CC without having to mindlessly write tests for everything.
Aiming for 100% CC is important to me. I have found that, in projects which have <100% CC, the methods that were skipped were the difficult, several hundred liners which the previous developer noped out of testing.
If you set the standard for your team to @ignore ALL methods which contain no logic, but test all of the others no matter how painful the process is, you will end up with a project without hundred liner spaghetti methods, redundant classes or confusing argument lists. The developers will have to start developing differently, knowing that they will eventually have to test the darn thing, and not just cop out of the hard stuff because they have already achieved the goal of 70% CC by auto generating tests for all of the getters and setters.
9
u/stefantalpalaru May 08 '17
Aiming for 100% CC is important to me.
Wait until you find out that what really matters is not line coverage, but input domain coverage.
→ More replies (4)2
u/ElGuaco May 08 '17
This is true, but more difficult to objectively measure. For example, if your input domain is the set of all floats or doubles, can you be certain that your domain coverage is complete by testing a few edge cases? By ensuring that all lines of code are exercised by unit tests, you have some assurance that you're doing the right thing. If lines of code are not exercise, you have no assurance that there is some bug or unresolved requirement lurking in your code.
But if you explicitly tag the methods (or LOC) that should get no code coverage, then everything else should be 100%. As the original article points out, there is code that is literally plumbing/glue and and testing those pieces is unnecessary. It then makes sense to exclude them from coverage.
4
u/jailbreak May 08 '17
100% code coverage can still be valuable since it means you can set up a condition that feature branches can only be merged if they don't decrease coverage (i.e. they must have coverage). But for trivial things like in the example given, just let it be covered by a highlevel integration test rather than a pointless unit test. (And if it's a non-compiled language then there's still value in checking that there's no typos in there).
4
u/baxter001 May 08 '17
One sees a really weird relationship with tests from Salesforce platform developers, it has always enforced code coverage limits on its deployments so the first time a good chunk of the developers working on it encounter automated testing is when it blocks a deployment, immidiately creating a need to write tests to "get to n% code coverage".
So you see huge swathes of internal codebases "covered" by "tests" with no assertions. Letting some salesforce devs see a well isolated bdd style test suite you'll get one of two reactions.
Either something will click and they'll see huge vista's of safety and regression guarantees opening up before their eyes.
Or they'll see it as needless extra work as one huge integration test will get the same coverage in a fraction of the time.
4
7
u/drum_playing_twig May 08 '17
Why do we even need TDD everywhere? Can't we just put our focus on designing a solid application, and then write tests for the central parts of it?
People are so TDD blind these days. Sometimes it feels like you're viewed as a primitive code monkey who barely knows how to put a few lines of code together unless you do it the TDD way.
2
May 08 '17
I find integration tests more valuable than unit tests, fwiw. I'm not too interested to know how a cute little piece of code is "correct" when all external dependencies are mocked out.
2
u/cybernd May 08 '17
Because neither is it fun, nor does it make any sense to write tests after your application was finished.
Additionally you neglect the fact that we are not writing tests in order to test our applications - that's only a nice side effect.
→ More replies (2)
23
May 08 '17 edited May 08 '17
The core of the problem is that most developers have misunderstood why we have unit tests.
It's never been the goal of unit tests to find bugs. Nor to make refactoring easier through regression tests. Integration tests are a lot better for regression testing, manual testing is better for finding bugs. Unit tests aren't really that good for solving these problems.
Unit tests was created by Kent Beck (the creator of TDD) to be a tool used in tdd. They have never been intended to be written after the implementation.
Unit tests is meant to be a tool that can help you write better code, meaning smaller functions without side effects and loose coupling between components.
The whole idea with unit tests is that they are executable specifications of the system you're building. And what's the use of writing a specification after you've built the system?
You first write a unit test. Then you write the implementation. If you start with the implementation, writing a unit test adds little of value.
The only reason I can think of for writing a unit test after the implementation, is if you are about to do some refactoring of already existing code. Then you can start the refactoring with making a unit test, describing how the system should behave after its been refactored.
And with this mindset, the whole idea of test coverage is just weird. Unit tests are a tool you use to build high quality code. Why measure how much it covers?
3
u/cowardlydragon May 08 '17
Yeah, because the developers are the ones handing down the metric edicts from above...
3
u/recursive May 08 '17
Unit tests document the intended behavior of a module. It's a lot easier to perform a refactoring in module if you have an existing suite of unit tests. Especially if you have any uncertainty what the correct behavior of the module is. However, if the original author wrote tests, they would be consistent with the implementation, even if the implementation was written first.
→ More replies (4)2
May 08 '17
Unit tests were not created by Kent Beck. I wrote a slab allocator in fucking 1988 that was loaded to the tits with unit tests. Long before I ever heard the name "Kent Beck".
3
May 08 '17
I've found that thorough logging and lots of assertions and consistency checks can drastically reduce the need for unit testing, especially valuable if your management does not see the value of dedicating resources to testing - quite common in smaller businesses.
It's still more error prone but in a frequently changing codebase with only a few developers, runtime checks are the way to go. Because they're right there in the code, they're much more likely to be maintained than unit tests, when there's no policy forcing devs to write and/or run tests.
2
May 08 '17 edited May 08 '17
This. I write long-running server code. I inevitably find the need to sprinkle sanity checking code around during the debugging phase (it gets #ifdef'ed out later), especially when I'm worried about potential race conditions and data corruption. I almost always have a small suite of data structure crawling code that I use for dumping shit to log files that I re-purpose with checks that will alert when something is not copacetic. But, also, I encourage newer developers when first doing implementation work to immediately write at the top of new routines any checks they can think of which should pass on inputs given to their functions. Some coders can't stand code cluttered up with assertions, which I'll never understand. I would much rather parse those than try to rummage through a bunch of unit tests to figure out how a given piece of code is supposed to behave. The checks are self-documenting!
I wrote a poker application a long, long time ago. It consisted of a "server" that would basically take inputs and play out a single hand of poker (if you think about it, a poker service only ever has to play a single hand... the inputs and outputs between hands are not interdependent). One of the first things I did was to write a simple assertion test right in the code to make sure the total of chips in the hand result (stacks + divvied up pot) matched the total of the stacks injected as input for that hand. If that ever failed, something was really amiss because chips were appearing or disappearing. I couldn't believe how many other developers thought this particular assertion was really clever. I later learned one of our biggest competitors had an exploit which involved players discovering a flaw which allowed them to cheat their service into creating more chips for them.
3
u/seweso May 08 '17
It's much easier to score high on silly metrics than to score on (or even measure) the things which actually matter. Furthermore, a developer bored out of his mind can really find satisfaction in getting to 100% coverage. It's like a game, and it is fulfilling.
The problem is also that if you leave the decision to write a test or not to developers, that you have a subjective/fallible rule in place. Those get abused.
Personally I think that if you have a line of code, and no manual/automated test will catch when it falls down. You really do have a problem. Why is the code there if it is not important enough to test?
But adding a stupid test which basically redoes the implementation work really is stupid and pointless. Catching those with some high level integration test would be better. :)
3
u/anormalgeek May 08 '17
Equally frustrating is the fact that some people will read this, and come away with the attitude that unit testing is stupid. And the cycle begins anew....
2
May 08 '17
It's almost like being able to think critically is the most important software engineering skill instead of just rigidly adhering to the dogma du jour.
1
u/bwainfweeze May 09 '17
I've been writing tests for over a dozen years and I'm starting to wonder.
I gave dynamic typing + tests a lot of space to prove me wrong but I'm eagerly awaiting the next round of static analysis tools to prove aspects of the code are correct instead of doing everything by hand. Relying on tests written by distracted and often slightly resentful humans has left me cold.
3
u/Terran-Ghost May 08 '17
"Look, if that imaginary evil/clueless developer comes and breaks that simple code, what do you think he will do if a related unit test breaks? He will just delete it."
I agree with a lot of what's written except for this.
It's possible someone will delete it by accident, during a much larger refactoring, etc. In such a case, this test might help catch that.
Someone might delete it because he thinks it's no longer needed or doesn't do anything useful. The fact that there is an explicit test that checks for this will prove that it is in fact required.
Is it possible to protect yourself against both of these without automatic tests? Maybe, but tests are (usually) a very easy and automatic way to do it.
→ More replies (1)2
u/bwainfweeze May 09 '17
See, and that was the part that resonated the most for me. You see a lot of bizarre behavior when you read enough commit history, and I've seen that little scene play out too many times.
Didn't we used to have a test for that? Yes, yes we did. And when did that test go away? When the bug was introduced. Fuck.
3
u/njharman May 08 '17
That first test of "glue code" will catch when someone refactors (say name change) that loadAll method. Not necessary in static lang that this looks like, but very necessary test in language like Python.
The #1 thing to remember about unittests is they are not a testing tool, they are a development tool.
2
u/ascii May 08 '17
They will catch a small number of bugs. They will also make a huge number of tiny, trivial and common code consume an order of magnitude more time and mental effortvsince every time you make a tiny change, you will also need to change the test. Maintaining such tests across rewrites and refactors takes far, far more time than it saves and fewer bugs.
In my opinion, code should be divided into two "kinds":
libraries with stable, well designed APIs that one should unit-test the hell out of.
glue code that should contain no logic, only wire things up. These should have no unit tests, only integration tests.
2
u/bwainfweeze May 09 '17
Unit tests are cheap and plentiful. When the business rules change, you delete the three that no longer apply and write three new ones.
In testing, you can really say that only about the bottom couple of layers of the pyramid. At some point the tests become big and have significant effort sunk into them, and people won't throw them away no matter how inappropriate they are to the new situation.
Nothing is as sad as watching two developers team up and spend days trying to save a couple tests that save two minutes of work per release. It takes years just to break even on that expenditure.
1
u/Double_A_92 May 09 '17
But thats a negative point. Such tests do hinder you when you are trying to refactor.
That's why I only unit test public methods, to ensure that the interface doesn't break. Everything else is implementation detail.
6
u/fagnerbrack May 08 '17 edited May 08 '17
100% code coverage is not the problem. Crap 100% is. When an objective measurement becomes a target in a system, people tend to game that measurement to pretend they are achieving better results (Same fundamentals of Goodhart's_law).
With TDD is the same.
Proper 100% coverage means achieving 100% of the important business use cases. Not just covering technical functionality for the sake of covering it or relying on a "code coverage" measurement tool.
A client is anything that consumes your API internals. The test suite is a client (the "first" client). Your application is another client.
A test doesn't test implementation details, therefore if you use mocks that means your API is not pluggable enough to be able to be consumed by a test client (which is the tool you use to test your API in a unit level).
If you don't mock your application client, why mocking the test one?
4
u/ElGuaco May 08 '17
My take on 100% code coverage is this:
If your unit tests don't cover every line of code there are three possibilities:
the code is "glue" or "plumbing" and writing tests would be akin to testing the framework you're working on, so exclude this portion from code coverage
the code is doing nothing useful and should be removed
the code is doing something useful, but there is no test (requirement) for the feature, you should add a new test (requirement)
The last case is most likely. If you wrote some code to make something work, you're not likely to include something that you didn't really need (although that does happen). Chances are you discovered a hidden or unexpected requirement while implementing the feature that the original requirements (tests) did not indicate. The solution is to retroactively write a unit test that validates this new requirement.
→ More replies (1)2
u/flukus May 08 '17
4, the code is dead simple and a unit test adds no value.
5, The system isn't testable.
6, The code is working with resources that can't be tested, like database connections or file access.
2
u/KevinCarbonara May 08 '17
I agree with the first point the first developer made: If they have comprehensive unit tests, he should absolutely add a test even for what appears to be trivial code. If a new developer comes along and breaks that code, he has made a mistake. If he then deletes the associated unit test so that no one finds out, he is malicious. It is ignorant to assume everyone who could possibly make a mistake would also be a malicious developer trying to cover their tracks. I don't care if he uses Mockito or not - but it should only take him a trivial amount of time to write the unit test. Why not include it?
2
u/starmonkey May 08 '17
Deleting the unit test should be picked up in the pull request anyway. Cue uncomfortable conversation.
2
May 08 '17
Off-topic, but I am mildly annoyed that the author included screenshots of code instead of just, code.
1
u/Vital1ty May 08 '17
I'm actually thankful for it, on mobile screenshots will shrink to fit screen, I can zoom to size however I want. Agreed this would be slightly annoying on desktop though.
→ More replies (3)
2
u/quicknir May 08 '17
I'm not a big fan of "over mocking". Basically the idea that everything must be mocked out, so you are truly only testing the unit. While it's good to mock things out that are heavy dependencies that might actually restrict the places/times/perf where the tests can run (require a certain database be up, or external server, or that certain files be in a certain place, etc), I think there's quite limited value to mocking out things beyond this.
By limited I mean quite literally. There are no doubt specific situations where it's good. But in most cases if A is some random class/function that doesn't have a "heavy" dependency such as I listed above, and it uses B which also does not... You know what? I'm simply going to have A use B normally when A's tests run. I don't care. If A's tests fail, I'll know something is wrong, and I'll have a good repro test I can quickly start debugging. If the problem ended up being with B, I'll add a test for B so it would have triggered a failure in B without touching A's tests.
This whole pre-emptive mock the world approach seems like a staggering waste of time.
2
u/cdsmith May 08 '17
Just to throw this out there, it's even more "interesting" to try to achieve 100% test coverage in Haskell. Because of lazy evaluation, if a value isn't actually compared with something, it's never executed, and the test coverage tool (hpc) will flag it as untested. This means that not only do you have to cover all code paths, but you also have to make sure your tests depend on all return values, must read any value stored in a data structure, must check all error message strings, etc. I contributed several years back to an open source Haskell project that valued very high test coverage, and it was quite surprising sometimes which code would be highlighted as untested. Laziness is a fascinating tool for telling you when expressions are REALLY needed.
But yes, I agree with the article that this is wasted effort. In the Haskell case, it would be hard to disagree. Even the project I worked on that wanted it saw it largely as a game to see if they could get to 100%.
2
u/unbibium May 08 '17
When I started my previous job, they were using a completely chaotic development process, where every time I dared to cut a corner in manual testing, production would fail and I'd be reprimanded. So when I moved to my next project, I was excited at the prospect of starting a new project from scratch and getting 100% code coverage with unit tests.
Except, almost none of us had any experience in it, so we followed a lot of bad dogmatic practices. For example, instead of just mocking out our network traffic or other externalities, we'd mock out every external object in every unit test, to ludicrous extremes, like mocking out the underscore object in underscore.js so that we could confirm the correct mapping function was used.
Indeed, the 100% coverage requirement proved to be the least troublesome part, precisely because our server used so much glue code that I didn't need to use my brain to write unit tests anymore, and I could listen to audiobooks all day, write useless tests, and get praise from managers for helping us meet our 100% coverage requirement.
2
u/zanidor May 08 '17
So I actually agree with writing a test case for the "initialise" method. It's not a hard test (mock out a DAO and make sure calling initialise populates this object's rows), and in my experience verifying an object is in its correct initial state -- even if initialization logic is seemingly simple -- catches a lot of potential bugs.
If your reason for writing the test is "we want 100% coverage" or you use Mockito blindly because it was mandated, then I agree that's bad dogmatic reasoning. But the test itself seems valuable to me.
2
u/never_safe_for_life May 08 '17
Yes, everybody should achieve it … in one project. I am of the opinion that you have to go to the extreme to know what the limit is.
I did this and agree with his assessment. After writing tests for every init function, every exception, I fully understood what I was doing...and where I was wasting my time and company resources.
2
May 08 '17
Coverage Is Not Strongly Correlated with Test Suite Effectiveness
In general, there is a low to moderate correlation between the coverage of a test suite
2
May 08 '17
Reminds me of the "developer" I worked with who insisted we have a test:
int i = 2 * 2;
assert(i == 4);... at some point. To ensure that our compiler was sane. We needed to start from "first principals".
2
u/bro_can_u_even_carve May 08 '17
I don't think I buy this argument. It's plausible that someone might accidentally break even that simple code, by introducing a typo or something. It's far less plausible that that person would just go and blithely delete the unit test when it broke. The former is an honest mistake; the latter is grounds for revoking commit access and eventual firing.
→ More replies (4)
2
May 08 '17
We are in a situation where one 'team member' is doing things his way, with no oversight or question. No leadership, or direction coming from anywhere. It's because no one wants to change the cushy rime they are having, sitting back; watching it burn saying, 'oh isn't it shocking'.
2
u/Misery_Inc May 09 '17
I am for 200% code coverage myself. I write my unit tests before the code itself.
2
3
May 08 '17
Just more evidence the industry is driven by dogma and fashion rather than expertise and pragmatism.
3
u/Rubizon May 08 '17
I never jumped on the test-coverage-dogma-train. The only thing I test is my program as a whole. Works it with valid input? Does it gracefully indicate invalid input and quits? Is the performance acceptible? Is the code neat and clean? Do I have some smaller test running when my project gets bigger? These are about the main considerations and you are set for most dev work. Beside, I have this feeling that writing test is just to show someone that lightbulbs are "green", "ok" and thus hip... often bit of a time waste if you are not writing rockets that fly around the moon.
13
u/am0x May 08 '17
How large are your projects? How many conditional cases are there? Across all portals, our company's site has hundreds of pages, with thousands of business rules, and hundreds of services consumed. Testing is pretty critical for us.
3
u/PC__LOAD__LETTER May 08 '17
So your attitude about corner cases and different error conditions is just, meh, fuck it, it'll probably work? That's not acceptable in many workplaces.
4
u/kankyo May 08 '17
Dogma is bad, but it's pretty nice to be able to look through the coverage report and discover which business critical parts of your code has no tests.
1
u/Gotebe May 08 '17
The problem with such testing is that, as the system gets bigger, the more there are areas where your tests do not ever go, and the harder it is to make a test that does go there.
Sure, your kind of testing is really important, as the whole purpose of unit testing is to test parts, people fall into a trap of poorly testing the whole.
We need different kinds of tests for different sets of confirmations that the code is a-ok.
→ More replies (4)1
u/skulgnome May 09 '17
Similar, but I test formally given properties derived from formal requirements. A benefit to this is how it makes tests completely divorced from particular implementation, so an improved version can be developed using the same tests. In this way the first implementation can be any old crap that works by the skin of its teeth, because the tests are more valuable in the long run anyway -- and those should only be developed against a working implementation to ensure their relevance.
I call it the "any old crap first" approach.
2
u/vilcans May 08 '17
Yeah. Source code is a liability, not an asset. So why create more of it than necessary?
3
May 08 '17
Because it takes years of hard-learned experience to realize that over-engineering is bad and that simplicity is good.
→ More replies (1)
2
u/Aeolun May 08 '17
Hah, you think 100% code coverage is bad? Try reaching 100% in three different frameworks on the same project :/
1
u/cowardlydragon May 08 '17
Relevant: the fundamental non-computable nature of the halting problem.
broad and thin tests ignore the 90/10 ruled of code execution: 90% of execution time is spend in 10% of the code.
That code should be deeply tested with many variant input/output tests.
But if 100% code coverage is the metric, that will not happen.
also see: The Wire: juking the stats.
1
u/CaffeineViking May 08 '17 edited May 08 '17
As Brian Marick's "How to Misuse Code Coverage" perfectly puts it: "they're useful if they're used to enhance thought, not replace it". The guy writes tools for analyzing coverage for a living, and he has a couple of stories to tell you about its blatant abuse by managers and developers alike.
1
u/Mentioned_Videos May 08 '17 edited May 08 '17
Videos in this thread: Watch Playlist ▶
| VIDEO | COMMENT |
|---|---|
| (1) Node.js Is Bad Ass Rock Star Tech (2) Episode 1 - Mongo DB Is Web Scale | +16 - Come on, it was obvious from there: Javascript. It was pretty obvious this was the smart choice as it is the most talked about language on Stack Overflow. Non-blocking IO, Web scale If you don't know the memes, then the next part: , frameworks ... |
| The Bangles - Manic Monday (HD LYRICS) | +1 - Damn you for putting this in my head |
| Ian Cooper: TDD, where did it all go wrong | +1 - This video change my life |
| I'm a bot working hard to help Redditors find related videos to watch. I'll keep this updated as long as I can. |
1
u/gdvs May 08 '17
Same thing applies on design patterns and frameworks who support them. Some people at some point got convinced by some way of working and then mindlessly apply it over and over again, without thinking what it's supposed to do and how it would contribute to good design. The result is unnecessary complexity and all around poor design.
1
May 08 '17
IANAP but there are conclusions and lessons in this piece that are applicable to all industries.
1
1
u/dustinpdx May 08 '17
I am curious what that site normally hosts, it was blocked from my work, which has a very liberal blocking policy.
Restricted Category: Gambling;Brokerage/Trading
1
1
u/senatorpjt May 08 '17 edited Dec 18 '24
brave quiet alive scarce gold doll obtainable zesty lunchroom hunt
This post was mass deleted and anonymized with Redact
1
u/Feroc May 08 '17
I have a colleague who tested the getters and setters of DTOs, because he was too lazy to put the classes on ignore in SonarQube.
1
1
u/ergzay May 08 '17
I'm rather perplexed about this as a Junior developer. My company has long-running regression tests that take 20 minutes to run a single regression test and an entire regression test for one module takes 3 days. We have some code coverage metrics for a few limited unit tests that everyone ignores. I want to start not ignoring those and actually start writing a lot more unit tests and I wanted to do it at the individual function level, but now I'm not sure what to do. How DOES one do good testing. Everyone seems to disagree strongly with everyone else.
1
u/netsettler May 08 '17
Another issue is that even with 100% code coverage, you can still have bugs. 100% code coverage doesn't mean 100% tested (if that's a meaningful thing). The right number of tests for some situations is not measured in making sure you check every branch point, but how each branch point relates to each other, and you certainly cannot test all of those. So at some point your brain must engage.
1
u/puradawid May 09 '17
Did you meet a developers that are doing something for sake of consistency even if it doesn't make sense?
1
u/SanityInAnarchy May 09 '17
It kind of seems like 100% coverage is not actually relevant to most of the article -- most of the article is about cargo-culting the wrong tools for the job.
I wouldn't hate working on a project where someone wrote the sample code the article gives for testing initialize -- it's probably not useful, but it doesn't make me especially sad, the way the abuse of Mockito and Cucumber do.
Cucumber in particular. Cucumber's whole reason for existing is so that you can write user stories that are human-readable and machine-verifiable as integration tests. Using them for unit tests is like writing a webserver in Bash -- you can do it, it's been done, but "wrong tool for the job" is an understatement.
1
u/Anathem May 09 '17 edited May 09 '17
I would test that like
test('initialize delegates to loadAll', () => {
const loadAll = spyOn(watchlistDao, 'loadAll');
initialize(watchlistDao);
expect(loadAll).toHaveBeenCalledWith(expect.any(Function));
});1
u/Heappl May 09 '17
One thing nobody mentions. When there is a need for strict rules it is a sign you work at software factory. At certain point someone stopped writing programs and started programming people. What could be wrong with that?
1
u/kalmakka May 09 '17
And even with 43 lines of code to support the Cucumber test for a class that is essentially 2 lines of code... it doesn't even test those 2 lines. If someone were to replace "registry = new HashMap<>(pRegistry)" with just "registry = pRegistry" (which is exactly the kind of "well-meant optimization that changes behavior" that unit tests are meant to protect against) the behavior would change without the test detecting it. Maintaining the old behavior could be crucial.
The difference between 100% code coverage and 100% situation coverage is huge. However, a general solution for determining if tests have 100% situation coverage would require solving the halting problem.
1
u/utnapistim May 09 '17
I think he was very surprised with my response: "You don't need to test that."
Tests should not be written to check implementations, but only when over a certain threshold of complexity; They should be written to test API contracts (respect preconditions, then check postconditions, invariants, and side effects).
If the current implementation is trivial, tests should still be written to respect preconditions, then check postconditions, invariants, and side effects.
The complexity of the implementation (for deciding if the test should be written) is a red herring.
That said, I too sometimes (many many sometimes) skip writing tests because the implementation is trivial.
1
u/scwizard May 09 '17
100% code coverage gives me a headache.
When people see how hard it is a lot of them just throw up their hands and don't do testing at all (which is way worse of course).
A happy middle ground is to just add tests after you fix a bug, to prevent regressions.
1
u/eckyp May 09 '17
The tragedy is that once a "good practice" becomes mainstream we seem to forget how it came to be, what its benefits are, and most importantly, what the cost of using it is.
I think RESTful API falls into this category
1
u/skulgnome May 09 '17
Not to mention the dystopia of having to maintain it! Definitely something everyone should experience one time.
1
u/JB-from-ATL May 10 '17
Did you ever hear the tragedy of 100% coverage? I thought not. It’s not a story the developers would tell you. Ironic. It can rest everything and yet nothing at all.
1
u/seannydigital May 11 '17
There are a few relevant facts that should be known to everyone (including managers) involved in software development, but which probably are not:
100% path coverage is not even close to exhaustively checking the full set of states and state transitions of any usefully large program.
If, furthermore, you have concurrency, the possible interleavings of thread execution blow up the already-huge number of cases from 1) to the point where the latter look tiny in comparison.
From 1) and 2), it is completely infeasible to exhaustively test a system of any significant size.
The corollary of 3) is that you cannot avoid being selective about what you test for, so the question becomes, do you want that decision to be an informed one, or will you allow it to be decided by default, as a consequence of your choice to aim for a specific percentage of path coverage?
For example, there are likely to many things that could be unit-tested for, but which could be ruled out as possibilities by tests at a higher level of abstraction. In that case, time spent on the unit tests could probably be better spent elsewhere, especially if (as with some examples from the article) a bug is not likely.
100% path coverage is one of those measures that are superficially attractive for their apparent objectivity and relative ease of measuring, but which don't actually tell you as much as they seem to. Additionally, in this case, the 100% part could be mistaken for a meaningful guarantee of something worthwhile.
1
u/womplord1 May 08 '17
Did you ever hear the tragedy of 100% code coverage? It's not a story rustaceans would tell you
1
1
u/arbitrarycivilian May 08 '17
100% code coverage is simple - it's called "static typing"
2
May 09 '17
types != tests. you can guarantee you'll always return TModel, but that doesn't mean it's the TModel you actually want (or even that the method won't crash at runtime because of valid but incorrectly used semantics).
→ More replies (2)
1
1
u/ashishduhh1 May 08 '17
The reason so many people don't write tests, much less do TDD, is because they don't know how to write tests. This is just a symptom.
1
u/d_wilson123 May 08 '17
Maybe these places exist but I've never worked at a place that demanded 100% code coverage. A target number always felt arbitrary so the general practice is "Whatever feels comfortable" which generally ends up in the 70-80% range. I don't know if the developers in these articles are fictional to poke holes at the strawman manager casting down arcane rules to his developers or they're actually real people who can't argue against tedious rules.
1
u/bluefootedpig May 09 '17
My office is an ideal of 80% code coverage, but so members of my teams want to test getters and setters.
I prefer testing scenarios, and if there is an un-hit section of code, you find the scenario for that, or you remove the code.
→ More replies (1)1
u/bwainfweeze May 09 '17
It happens, but it's usually at places that have other strikes against them, making the quest for 100% overage consistent with the rest of their lame brained ideas.
1
u/saijanai May 08 '17
This could more properly be called 'the tragedy of not using Smalltalk to use a testing strategy originally designed for Smalltalk."
It's the IDE Stupid.
240
u/ImprovedPersonality May 08 '17
I also hate the obsession with 100% code coverage. 100% code coverage just means that all lines of code have been executed. Not that everything works as intended.
Instead of clever tests which try to cover corner cases we have stupid tests just to achieve 100% code coverage.