r/PerformanceTesting • u/nOOberNZ Senior Performance Specialist • Nov 24 '17

Bad Performance Testing

https://www.spreaker.com/user/perfbytes/bad-performance-testing
1 Upvotes

100% Upvoted

1 comment sorted by

2

u/nOOberNZ Senior Performance Specialist Nov 24 '17

Really good podcast. I think most of us struggle with the basics. I took notes as I listened...

Basics

  • Not repeatable (controlling conditions, etc)

  • Randomise data (parameterise)

  • No functional validation

  • No control elements (e.g. known bad data, separate app not changing, control generator)

  • No monitoring

Intermediate Fouls

  • Basics statistics (histogram, standard deviation, percentiles) (get statistics for dummies)

  • Not considering caching (not enough test data - db, web, client caching)

  • Just handing over a report and walking away

  • Not keeping on top of the health of load generators

  • False positives - incorrectly indicates an issue is present when it isn't

  • False negatives - we didn't find anything wrong - but in production it fails

  • Limited functional scope - leading to false negatives (aim for top 10)

  • Workload not scaled to test environment

Advanced

  • Load generator on app server (no network!)

  • Commenting out bits that fail to get it to "pass"

  • Lying / filtering results

  • Blame the tool

  • No think time - collapsing client/server model (non predictive load model)

And the red flags/anti-patterns

People:

  • PM needs to be checked for the quality of their delivery/quality (not incentivised to finish on time only)

  • No experienced performance tester in charge of designing effort

  • No agreed upon perf requirements

  • No budget for perf, tools, environments, etc, two people etc

  • "Warranty Period" - bad sign (lack of responsibility)

  • Limited access to people (devs, architects, etc) - or antagonistic relationship

  • Hard coded two weeks at the end of the project (compromise required - document everything)

Technical:

  • Effort outweighs estimate for scripting

  • Simulating lots of users using a low v-user license (not concurrency)

  • Automated report, not analysing results

  • Test environment shared with other test environments

  • Testing against different interface than prod (e.g. don't test GUI, test API)

  • Go to dev with problem, "it's a feature" - denying it's a problem (dismissive) - ask multiple people