Philosophy of Testing

Philosophy of Testing

1. Myers' 10 Principles

   • a test case must define expected output or result
   • programmer/organisation shouldn't test own programs
   • inspect results of each test
   • expected/unexpected and valid/invalid inputs must be tested
   • test to see if it does what it shouldn't, as well as if it doesn't what it should
   • Assume errors will be found
   • Make every test count
   • errors cluster

2. Meyer's 7 Principles

   • Definition: to test is to try and make fail
   • Tests do not replace specs
   • Any fault must yield a test case, to prevent it coming back
   • Automatically determine success or failure of tests, don't manually search
   • A failed test should mean a broken contract and vice versa
   • Should have manually and automatically produced test cases
   • Compare testing strategies using explicit criteria and in reproducible situtions
   • Best category to use in comparisons of test strategy is number of faults per unit time

3. Debugging

   • Treat debugging as a series of experiments
   • test hypotheses rather than make random changes
   • strategy
     • task is to find error, not disguise symptoms
     • first think about problem
     • then search and experiment
     • print statements
       • no special tools required
       • often badly used, don't just randomly print stuff
       • again, form hypotheses and test them with a print statement
     • Walkthroughs
       • Look at code line by line - looking at state changes
       • Explain it to someone else
       • Flexible ways of doing it, pencil and paper, on screen, with debugger
   • Debuggers
     • Breakpoints
       • test state
     • Can step-over/into routines
     • Call-sequence (Stack) inspectors
     • Object and variable state inspectors
     • But they are both language and environment specific
   • Sleep on it!