Chapter 8: Software Testing (3705)

Chapter 8: Software Testing (3705)

1. 8.1 Development Testing

   • Development testing includes all testing activities that are carried out by the team developing the system.
   • There are three stages of development testing:
     • Unit testing
       • Unit testing is the process of testing program components, such as methods or object classes.
       • Every object method should be tested
       • Generalization or inheritance makes object class testing more complicated. Inherited methods might not work as expected in subclasses
       • An automated test has three parts:
         • A setup part, where you initialize the system with the test case, namely, the inputs and expected outputs.
         • A call part, where you call the object or method to be tested.
         • An assertion part, where you compare the result of the call with the expected result
       • Testing is expensive and time consuming, so it is important that you choose effective unit test cases.
         • Effectiveness, in this case, means two things:
           • The test cases should show that, when used as expected, the component that you are testing does what it is supposed to do.
           • If there are defects in the component, these should be revealed by test cases.
       • Two strategies that can be effective in helping you choose test cases are:
         • Partition testing
           • equivalence partitions or domains
             • Eg. All positive numbers plus another positive = a positive number
           • Output equivalence partitions are partitions within which all of the outputs have something in common.
           • Make use of inputs at the partition boundaries
         • Guideline-based testing
           • Guidelines and Past experience
       • Whittaker’s book (Whittaker 2009) includes many examples of guidelines that can be used in test-case design
         • ■ Choose inputs that force the system to generate all error messages: ■ Design inputs that cause input buffers to overflow. ■ Repeat the same input or series of inputs numerous times. ■ Force invalid outputs to be generated. ■ Force computation results to be too large or too small.
     • Component testing
       • Software components are often made up of several interacting objects
       • There are different types of interface between program components and, consequently, different types of interface error that can occur:
         • Parameter interfaces
         • Shared memory interfaces
         • Procedural interfaces
         • Message passing interfaces
         • Interface errors are one of the most common forms of error in complex systems (Lutz 1993). These errors fall into three classes:
           • Interface misuse
             • A calling component calls some other component and makes an error in the use of its interface
           • Interface misunderstanding
             • A calling component misunderstands the specification of the interface of the called component and makes assumptions about its behavior.
           • Timing errors
             • in real-time systems that use a shared memory or a message-passing interface
         • Some general guidelines for interface testing are:
           • Examine the code to be tested and identify each call to an external component
           • Where pointers are passed across an interface, always test the interface with null pointer parameters
           • Use stress testing in message passing systems
           • Where several components interact through shared memory, design tests that vary the order in which these components are activated.
     • System testing
       • System testing during development involves integrating components to create a version of the system and then testing the integrated system
       • checks that components are compatible, interact correctly, and transfer the right data at the right time across their interfaces
       • overlaps with component testing, but there are two important differences:
         • During system testing, reusable components that have been separately developed and off-the-shelf systems may be integrated with newly developed components. The complete system is then tested.
         • Components developed by different team members or subteams may be integrated at this stage
       • All systems have emergent behavior
       • Because of its focus on interactions, use case-based testing is an effective approach to system testing
         • The sequence diagram helps you design the specific test cases that you need, as it shows what inputs are required and what outputs are created:
         • A complete use case/scenario test must take these exceptions into account and ensure that they are correctly handled.
   • Development testing is primarily a defect testing process, where the aim of testing is to discover bugs in the software.

2. 8.2 Test-driven development

   • Test-driven development (TDD) is an approach to program development in which you interleave testing and code development
   • You develop the code incrementally, along with a set of tests for that increment.
   • Test-driven development was introduced as part of the XP agile development method
   • may be used in both agile and plan-based processes.
   • The steps in the process are as follows:
     • 5. Once all tests run successfully, you move on to implementing the next chunk of functionality
     • 1. You start by identifying the increment of functionality that is required. This should normally be small and implementable in a few lines of code.
     • 2. You write a test for this functionality and implement it as an automated test. This means that the test can be executed and will report whether or not it has passed or failed.
     • 3. You then run the test, along with all other tests that have been implemented. Initially, you have not implemented the functionality so the new test will fail. This is deliberate as it shows that the test adds something to the test set.
     • 4. You then implement the functionality and re-run the test. This may involve refactoring existing code to improve it and add new code to what’s already there.
   • As well as better problem understanding, other benefits of test-driven development are:
     • Code coverage
       • In principle, every code segment that you write should have at least one associated test.
     • Regression testing
       • You can always run regression tests to check that changes to the program have not introduced new bugs.
       • One of the most important benefits of TDD is that it reduces the costs of regression testing. Regression testing involves running test sets that have successfully executed after changes have been made to a system.
     • Simplified debugging
     • System documentation
   • If you are reusing large code components or legacy systems, then you need to write tests for these systems as a whole.
   • System testing also tests performance, reliability, and checks that the system does not do things that it shouldn’t do, such as produce unwanted outputs.

3. 8.3 Release testing

   • Release testing is the process of testing a particular release of a system that is intended for use outside of the development team.
   • Can be released for other teams or for the user
   • There are two important distinctions between release testing and system testing during the development process:
     • The system development, team should not be responsible for release testing.
     • Release testing is a process of validation checking to ensure that a system meets its requirements and is good enough for use by system customer
   • The primary goal of the release testing process is to convince the supplier of the system that it is good enough for use.
   • The system is treated as a black box whose behavior can only be determined by studying its inputs and the related outputs.
   • Requirements-based testing
     • A general principle of good requirements engineering practice is that requirements should be testable.
     • the requirement should be written so that a test can be designed for that requirement
     • Requirements-based testing is validation rather than defect testing—you are trying to demonstrate that the system has properly implemented its requirements.
     • You normally have to write several tests to ensure that you have coverage of the requirement.
   • Scenario testing
     • approach to release testing whereby you devise typical scenarios of use and use these scenarios to develop test cases for the system.
     • In a short paper on scenario testing, Kaner (Kaner 2003) suggests that a scenario test should be a narrative story that is credible and fairly complex. It should motivate stakeholders; that is, they should relate to the scenario
     • When you use a scenario-based approach, you are normally testing several requirements within the same scenario.
   • Performance testing
     • Once a system has been completely integrated, it is possible to test for emergent properties, such as performance and reliability.
     • involves running a series of tests where you increase the load until the system performance becomes unacceptable
     • performance testing is concerned both with demonstrating that the system meets its requirements and discovering problems and defects in the system
     • Experience has shown that an effective way to discover defects is to design tests around the limits of the system.
     • Stress testing helps you do two things:
       • Test the failure behavior of the system
       • Reveal defects that only show up when the system is fully loaded
       • Stress testing is particularly relevant to distributed systems based on a network of processors.

4. 8.4 User testing

   • User or customer testing is a stage in the testing process in which users or customers provide input and advice on system testing
   • There are three different types of user testing:
     • Alpha testing, where a selected group of software users work closely with the development team to test early releases of the software.
       • Alpha testing is often used when developing software products or apps
     • Beta testing, where a release of the software is made available to a larger group of users to allow them to experiment and to raise problems that they discover with the system developers.
       • Beta testing is mostly used for software products that are used in many different settings
     • Acceptance testing, where customers test a system to decide whether or not it is ready to be accepted from the system developers and deployed in the customer environment.
       • Acceptance testing is an inherent part of custom systems development.
   • Figure 8.11 shows that here are six stages in the acceptance testing process:
     • Define acceptance criteria
       • The acceptance criteria should be part of the system contract and be approved by the customer and the developer
     • Plan acceptance testing
       • deciding on the resources, time, and budget for acceptance testing and establishing a testing schedule
       • required coverage of the requirements and the order in which system features are tested.
       • define risks to the testing process such as system crashes and inadequate performance, and discuss how these risks can be mitigated.
     • Derive acceptance tests
       • Once acceptance criteria have been established, tests have to be designed to check whether or not a system is acceptable
     • Run acceptance tests
       • The agreed acceptance tests are executed on the system
       • Ideally, this step should take place in the actual environment where the system will be used
     • Negotiate test results
       • It is very unlikely that all of the defined acceptance tests will pass and that there will be no problems with the system
     • Reject/accept system
       • This stage involves a meeting between the developers and the customer to decide on whether or not the system should be accepted

5. Introduction

   • Testing is intended to show that a program does what it is intended to do and to discover program defects before it is put into use.
   • When you test software, you are trying to do two things:
     • Demonstrate to the developer and the customer that the software meets its requirements.
     • Find inputs or input sequences where the behavior of the software is incorrect, undesirable, or does not conform to its specification
   • Testing is part of a broader process of software verification and validation (V & V)
   • Verification and validation processes are concerned with checking that software being developed meets its specification and delivers the functionality expected by the people paying for the software
     • Software verification is the process of checking that the software meets its stated functional and non-functional requirements.
     • software validation is to ensure that the software meets the customer’s expectations
   • The level of required confidence depends on the system’s purpose, the expectations of the system users, and the current marketing environment for the system
     • Software purpose
     • User expectations
     • Marketing environment
   • the verification and validation process may involve software inspections and reviews
     • Inspections mostly focus on the source code of a system,
   • Software inspection has three advantages over testing:
     • During testing, errors can mask (hide) other errors.
     • Incomplete versions of a system can be inspected without additional costs
       • Need specialized test harnesses to test the parts that are available when running code -> increases development costs
     • searching for program defects, an inspection can also consider broader quality attributes of a program, such as compliance with standards, portability, and maintainability
   • inspections cannot replace software testing
     • Inspection very good for seeing programming errors but not good at seeing how components interact with one another
   • Typically, a commercial software system has to go through three stages of testing:
     • Development testing
       • tested during development to discover bugs and defects
     • Release testing
       • separate testing team tests a complete version of the system before it is released to users
     • User testing
       • users or potential users of a system test the system in their own environment

Imagens e arquivos do mapa