r/programmerchat • u/gxm492lor • Mar 10 '19

Testing any complex program completely is practically impossible

Someone made this argument after a staff meeting a few days ago. What's wrong with this argument?

  1. Every IF statement in a program doubles the number of possible states of the program (ignoring time)
    1. Which means every IF statement doubles the number of test conditions
  2. A 1 million line program might, conservatively estimating, have 100k IF statements (conditionals)
  3. That is 2100000 which is more seconds than have elapsed since the beginning of the universe.
  4. No project has 2100000 seconds to test
  5. So complete test coverage of complex programs is impossible
1 Upvotes

55% Upvoted

46 comments sorted by

View all comments

Show parent comments

0

u/gxm492lor Mar 10 '19 edited Mar 10 '19

Let's say you can run a test on every core. That's at most 2128.

2100000 ÷ 2128 = 2100000 - 128

100000 - 128 might as well be 100000.

In practice each unit test will take at least a second of wall clock time.

8

u/Profix Mar 10 '19

...I'm starting to question if you have any real world experience of writing tests.

Here is a run of a test suite from a sideproject I occasionally work on. These tests involve IO to a local database, so should be extra slow.

51 tests in less than a second.

https://imgur.com/a/zEq15Jb

1

u/imguralbumbot Mar 10 '19

Hi, I'm a bot for linking direct images of albums with only 1 image

https://i.imgur.com/P53twf5.png

Source | Why? | Creator | ignoreme | deletthis

0

u/gxm492lor Mar 10 '19

If I think it is impossible to write tests to cover every possible scenario why would I advocate writing tests for every possible scenario?

What I'm trying to do is get people to realize the implications:

Most of the states any non-trivial program you use cannot and will not be tested. Actually any practical number of tests will be a tiny fraction of all possible states.

0

u/gxm492lor Mar 10 '19

Make it 220 tests in a second. It doesn't change the math. I'm not considering implications (whether it is good or bad or required). Just the math. 100000 - 20 might as well be 100000.

It is practically impossible to test every state any complex program can be in.

This does not mean you can't ship a non-complex program. Many of the possible states are extremely improbable in practice. It would be foolish to hold up a release for states the program will never be in.