1 - The Fundamentals of Testing

Página 1

THE FUNDAMENTALS OF TESTING - CHAPTER ONE

WHAT TESTING IS AND WHAT TESTING DOES

Testing makes a rigorous examination of the behaviour of a component or system and reports all defects found for the development team to correct.

Testing then repeats enough tests ensure that defect corrections have been effective.

Static & Dynamic Testing

Static testing is testing where there is no code, such as testing the requirement documentation

Dynamic testing is when we test an application using the app and it's some form of testing data, this can be referred to as test execution.

GENERAL TESTING PRINCIPLES

The purpose of testing is to find defects, testing can only show that one or more defects exist, it can't show that a program is errors.

Early Testing

In terms of cost, it is far cheaper to find errors at the earlier stages of the SLDC than at the later stages

Defect Clustering

About 80% of defects found in an application are found in about 20% of the modules. It is therefore useful if testing activity reflects this by targeting areas where a high number of defects have been found. There may be fewer defects in the other 80% but they should still be tested diligently for bugs.

FUNDAMENTAL TEST PROCESS

The fundamental test process consists of five parts that encompass all aspects of testing.

Planning & Control

Analysis and Design

Implementation & Execution

Evaluating Exit Criteria and Reporting

Test Closure Activities

Planning & Control

Test planning is determining what is going to be tested and how this will be achieved, it is also where the test completion criteria is defined.

Control is what we do when the testing activities don't match up with the plans. It's an ongoing activity where we compare the progress against the plan.

Typical Tasks Would Be

Defining the scope and objectives of testing and identifying risks

Determining the test approach

Detailing what is required to carry out the testing

Implementing test policies and strategies

Scheduling test analysis and design tasks

Scheduling test implementation, execution, and evaluation

Detailing when testing will stop ie the exit criteria

Measuring and analyzing results

Comparing expected and actual progress, test coverage and exit criteria

Making corrections if things go wrong and deciding actions

Typical Test Case Design Techniques

Reviewing requirements, architecture, design, interfaces and other parts which collectively comprise the test basis

Analysing test items, specifications, behavior and structure to identify test conditions and test data required

Designing the tests

Determining whether the requirements and the system are testable

Detailing what the test environment should look like and whether there are any infrastructure and tools required

Test Implementation and Execution

Evaluating exit criteria and reporting

Testing closure activities

Testing closure activities concentrate on making sure that everything is complete and tidied away, such as reports written, defects closed and deferred for another phase of testing

Typical Test Closure Tasks

Ensure all documentation is in order and up to date

Closing down and arching the test environment

Passing over test ware to the maintenance team

Writing down any lessons learned from the testing project.

THE PSYCHOLOGY OF TESTING

The aim is to work together rather than be confrontational, keep the focus on delivering a quality product.

Results should be presented in a non-personal way. The work item may have errors or be incorrect so point this out in a non-personal way.

Attempt to understand how others feel, it is possible to discuss problems and still leave others feeling positive

Confirm understanding after conversations & discussions

Página 2

LIFE CYCLES - CHAPTER TWO

SOFTWARE DEVELOPMENT MODELS

A typical development life cycle for a software product involves

Requirement Specification

Functional Specification

Technical Specification

Program Specification

Coding

Testing

The above steps are usually used in a waterfall model which is known as a linear or sequential model. In this model, all testing is carried out at the end when all coding has been completed where it will either pass or fail. One of the main issues with this model is if certain parts of the application fail, under normal circumstances it would be difficult to separate those parts from the rest of the application and only release the passed areas.

Verification

Checks that the work product meets the requirements set out for it. This helps to ensure that we are building the product in the correct way.

Validation

Changes the focus of work product evaluation to evaluation against the user needs. This means ensuring that the behavior of the work product matches the customer needs as defined for the project. This helps to ensure we are building the correct product as far as the end-users are concerned.

Development Models

There are two types of development model which facilitate early work-product evaluation

V Model

The V-Model is an extension of the waterfall model. The Left-hand side of the model focuses on elaborating the on the initial requirements.

Requirement specification - capturing user needs

Functional specification - definition of functions required to meet user needs

Technical specification - technical design of functions identified in the functional spec

Program specification - detailed design of each module or unit to be built to meet required functionality

The middle of the V-model shows that planning for testing should start with each work product.

The right-hand side of the V-model focuses on testing activities where for each work product a testing activity is identified

Testing against the requirements spec takes place at the acceptance testing stage

Testing against the functional spec takes place at the systems testing stage

Testing against the technical spec take place at the integration testing stage

Testing against the program spec takes place at the unit testing stage.

Each stage must be completed before the next one can be started, this approach software development pushes validation of the system by the user representatives right to the end of the life cycle. If requirements are not captured correctly or are changed, issues may not be uncovered until user testing is carried out. Fixing issues at this stage could be very costly and could lead to the project being canceled.

Iterative Development Models

In this model, the requirements do not need to be fully defined before coding can start. Instead, a working version is built in a series of iterations/stages where each iteration encompasses requirement definition, design, code, and test.

The project will have a timescale and cost with stages which have defined timescales and costs.

Forms of iterative development include prototyping, rapid application development (RAD) and agile software development.

TEST LEVELS

Characteristics of good testing across development life cycle include:

Early test design

Each work-product is tested

Testers are involved in reviewing requirements before they are released.

Typical Levels of testing are

Unit Testing

Integration Testing

System Testing

Acceptance Testing

Unit Testing

Generally, the code will be written in parts known as units or modules. Unit testing is intended to ensure that the code written for the unit meets the specification and that all code can be fully executed. Unit tests are usually performed by a developer and defects found tend not to be documented.

Integration testing

Once all units have been tested and passed they are integrated together. The next stage of testing is to test all of the integrated units. The purpose of integration testing is to expose defects in the interfaces and in the interactions between integrated components or systems.

Before integration testing can be planned, an integration strategy is required which involves making decisions on how the system will be put together before testing.

Big Bang Integration

This is where all units are linked at once, resulting in a complete system. It's difficult to isolate defects and is considered a bad choice as it introduces the risk that issues will be found late in the project resulting in high costs.

Top Down Integration

This is where the system is built in stages starting with components which call other components. Components that call other are usually placed above components which are called.

Each component must be tested when it is built. If a component to be called has not yet been integrated a stub may be used to replace it

Bottom Up Integration

This is the opposite of top-down integration. If top level components which will make calls have not yet been written, then drivers may be used to replace them.

Levels of Integration Testing

There may be more than one level of integration testing

Component integration testing - focuses on the actions between components and is completed after unit testing and is usually performed by developers.

System integration testing - focuses on the interactions between different systems and is completed after individual testing of each system. This type of integration testing is usually performed by testers.

SYSTEM TESTING

Once all components have been tested and are working at a unit & integration level, the next step is to consider the functionality from an end-to-end level.

System testing is required because testing at unit and integration level does not take into account interactions across the whole system and potential errors due to the environment.

System testing should be carried out in an environment replicating the live environment and would typically be carried out by a team which is independent of the development process.

In the V-Model the behavior required for the system is defined in the functional specification, which contains definitions for both functional and non-functional requirements.

Functional Testing focuses on what the system should do. Non-functional system testing looks at aspects which are important but not directly related to what functions the system performs. Non-Functional requirements tend to be generic requirements which can be applied to many different systems, such are responding to inputs in a reasonable time manner. Typically these requirements will consider both normal operations and behavior under exceptional circumstances.

Examples of non-functional requirements

Installability - installation procedures

Interoperability - operation of the application in different environments

Maintainability - ability to introduce changes to the system

Performance - expected normal behavior

Load Handling - Behaviour of the system under increasing load

Stress Handling - Behavior at the upper limits of system capability

Portability - Use on different operating systems

Recovery - Recovery procedures on failure.

Reliability - Ability of the software to perform its required functions over time

Usability - ease with which users can engage with the system

* Note - Security is regarded as functional requirements in the learning syllabus

ACCEPTANCE TESTING

The next step after system testing is acceptance testing. The purpose of acceptance testing is to provide the end user with confidence that the system will function according to their expectations. The key purpose of acceptance testing is to demonstrate system conformance to the customer requirements.

In the V-Model acceptance testing will be carried out using the requirements spec as a basis for testing.

Acceptance testing is usually the responsibility of the customer or users of the system.

Typical forms of acceptance testing include the following:

User Acceptance Testing - Testing by user representatives to check that the system meets their business needs.

Operational Acceptance Testing - Involves checking that processes and procedures are in place to allow the system to be used and maintained such as Backup files, Procedure for disaster recovery, Training for end users, Maintenance Procedures, Security Procedures.

Contract and Regulation Acceptance Testing -

CAT is when the criteria for accepting a system is documented in a contract. Testing will eventually be conducted to check that these criteria have been met before the system is accepted.

R-A-T Regulation acceptance is checking that the system meets regulations and legislative demands as required for systems built in with certain sectors such as government, defense, banking, pharmaceutical.

Alpha & Beta Testing -

Alpha Testing is testing that takes place at the site where the system was developed

Beta Testing is testing which takes place at the customer's site.

TESTING RELATED TO CHANGES

Confirmation testing is re-testing a part of the program where a defect has previously been found to confirm that the found defect has now been removed.

Regression testing is re-testing the unchanged software to ensure no other issues have been introduced during a bug fix. Regression testing should also be carried out following any change to the environment. Regression testing involves creating a set of tests which serve to demonstrate t the system works as expected. These tests are likely to be used many times, so regression testing is a suitable candidate to be for automation.

MAINTENANCE TESTING

Nearly all project will eventually be released to a live environment. During their deployment, there may be various changes required such as:

Additional features added

Migrations to new operating systems/environments

System being retired

New faults being found and fixed

Any changes made to the live system must be tested and ideally, the application should be subject to regression testing. It would likely be unfeasible and costly to regression test all areas, so it must be determined which areas are most likely to be affected by the new changes, this is referred to as impact analysis. Impact analysis can be difficult, as the original requirements may have changed or other issues such as the original developers are no longer available and an overall lack of documentation or out of date documentation exists.

Página 3

STATIC TESTING - CHAPTER THREE

Static Techniques

Static testing is a technique which tests around software without actually executing any code. This typically includes checking requirements and specification documentation.

The first of these is known as review and is typically used to find and remove errors and ambiguities in documents before they are used for development reducing cost and time.

The second is known as static analysis, which enables code to be analyzed for structural defects or systematic programming weaknesses that may lead to defects. This is usually completed using automation tooling.

Reviews

Reviews can be used to test against anything such as requirements, system designs, code and test plans. Reviewing code against development standards can prevent defects appearing in test execution, although as the code has already been written, not all potentials costs will be greatly reduced.

Important Benefits of finding defects early

Development productivity can be improved and time scales reduced.

Testing cost and time will be reduced by removing the main delays in test execution.

Reduction in lifetime costs can be achieved because fewer defects in the software means fewer support costs

Results in improved communication and clarity of requirements

Types of Defects typically found by reviews

Deviations from standards which have either been internally defined or managed by legislation and regulations.

Requirements defects - The design not matching the requirements

Insufficient maintainability - The code being too complex to maintain

Incorrect interface specs

Review Process

The formality of a review process can vary based on the following:

The maturity of the development process, the more mature, the more formal

Legal or regularity requirements

The need for an audit trail. A formal review process ensures that it's possible to trace back through the SDLC.

Reviews can have a variety of objectives with review objective defining the main focus of the review.

Finding defects

Gaining understanding

Generating discussion

Decision making by consensus

All reviews will exhibit the same basic elements of process

The document under review is studied by the reviewers

Reviewers identify issues or problems and inform the author either verbally or in the documentation.

The author decided on any action to take in response to the comments and updates the document accordingly

Phases of a formal review

Planning

Selecting the personnel

Allocating Roles

Defining the entry and exit criteria

Selecting which parts and which documents should be reviewed

Kick-Off

Individual Preparation

Review Meeting

What Time is available

Requirements of the author

Type of review

Rework

Follow-up

Roles and Responsibilities

The role of each reviewer is to look at the documents belonging to them from their assigned perspective. Checklists may be used which are based on that user's perspective such as user, maintainer, tester or operations.

Type of review

A single document may be subject to many different review types. An informal review may be carried out before the document is subject to a technical review or it could also be possible depending on the level of risk that a technical review or inspection may take place efore a walkthrough with a customer.

Levels of formality by review type

Informal

No formal process underpins the review

Maybe documented but is not required

Maybe variations in the skills of the reviewer

The main purpose is to find defects

May be implemented using pair programming (One dev reviews code of another) or reviewed by a technical lead

Walkthrough

Led by author of document and attended by members of authors peer group

Review sessions are open-ended

Technical Review

Inspection

STATIC ANALYSIS BY TOOLS

Static analysis is looking for defect without executing the code once. Using static analysis we can find defects that are hard to during test execution by analyzing the program code e.g instructions to the computer can be in the form of control flow graphs and data flows

The value of static analysis is:

Early detection of defects prior to test execution.

Early warning about suspicious aspects of the code or design

Identification of defects not easily found by dynamic testing

Improved maintainability of code and design

Prevention of defects

Typical defects discovered by static analysis tools include:

Referencing a variable with an undefined value

Inconsistent interface between modules and components

Variables that are never used

Unreachable (dead) code

Programming standard violations

Security vulnerabilities

Syntax Violations

Página 4

TEST DESIGN TECHNIQUES - CHAPTER FOUR

TEST CONDITIONS, TEST CASES, AND TEST PROCEDURES

The design of tests comprises three main steps

Identify test conditions - What we test for

Specify test cases - What is used to test with

Specify test procedures - Steps taken to carry out the test

Test Condition - An item or event of a component or system that could be verified by one or more test cases, e.g. a function, feature, quality attribute or structural elements. A test condition is some characteristic of our software that we can check with a test or a set of tests, it is the condition we would expect to see happen that we test for.

Test Case - A set of input values, execution preconditions, expected results, and execution, post conditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. A test case gets the system to some starting point for the tests, then applies a set of input values that should achieve a given outcome and leaves the system at some end point.

We should define here what state the system is in when we start and what state it is in after the test in order to check it ends up in the correct place.

Test Procedure Specification - A sequence of actions for the execution of a test. A test procedure identifies all the necessary action in sequence to execute a test, these can also be called test scripts or manual test scripts.

Test Execution Schedule - This is used to test the whole system by placing together each of the individual test procedures in a sequence of steps in order of highest priority to be executed. We should also think about ordering in terms of what makes the most sense when testing and group relevant tests together.

THE IDEA OF TEST COVERAGE

Test coverage provides a quantitative assessment of the extent and quality of testing, answering the question of how much testing was done.

There are two important reasons why we need to know about test coverage:

It provides a quantitative measure of the quality of the testing has been done by measuring what has been achieved

It provides a way of measuring how much more testing needs to be done Using quantitative measures we can set targets for test coverage and measure against them.

CATEGORIES OF TEST CASE DESIGN TECHNIQUES

The main techniques to learn here will be

Specification-based / Black-box Techniques - Deriving test cases directly from the specification or the model of a proposed system.

Structure-based / While-box Techniques - Deriving test cases directly from the code written to implement a system.

Experience-based Techniques - Deriving tests cases from the testers experience of similar systems and general experience of testing.

These are not the only categories available

SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES

The source of information on which to base testing is knowing as the "Test Basis", this will typically be a requirement spec of model spec. It the test basis is well defined and adequately structured we can easily identify test condition from which to test against.

If a detailed requirements spec is available it is better as a tester to write test cases using this, as it increases the chance of finding defects and you are testing directly against what was specified. If you build test cases against the already built program it becomes more difficult to identify what is an actual defect or what is a genuine feature enhancement.

If no detailed specification is available, a tester may need to speak with key stakeholders in order to determine the program's expectations.

It's also likely that a specification will contain nonfunctional requirements such as usability, reliability, and performance expectations.

Five Specification Base Techniques

Equivalence partitioning

Boundary value analysis

Decision table testing

State transition testing

Use case testing

Equivalence partitioning

Input Partitions

We can use input partitions to group the test conditions where we expect the program to pass or fail. If we had an input should only expect values -10,000 to 10,000, then the passing values could be split up into three separate partitions

Negative Valid Values - Positive Valid Values - Zero

This then allows us to choose a number to represent each of these groups and typically a number in the middle range would be chosen.

We would then need to consider invalid values and split those into various partitions also, however as with most test there will be a greater range of potential invalid values than valid, because of this we may need to limit our testing to using values which would be the most likely to crop up in a real world situation.

Output portions

If enough is known about an application we may be able to partition other values instead of just input/output values. If say a program handled input values by placing them in queues we could in principle check that the requests end up in the correct queue. In this, we could then create a number of partitions based on the correct queue.

Boundry Value Analysis

Boundry value analysis looks at the min & max valid values for some input. It's important to test and consider these as boundaries are an area where mistakes are typically made for various reasons such as a for loop making one less or more loop than it should have. If a program accepted a number with the range 1 - 100 then

1 would be lower valid boundary

100 would be the higher valid boundary

We would also have non-valid boundaries which in this case would be

0 would be a lower invalid boundary

101 would be a higher invalid boundary

We would then test each end of the boundary so could run tests against

Valid Test 0 1 2

Invalid Test 99 100 101

Decision Table Testing

Decision tables are used to capture all logical decisions. Specifications often contain business rules to define functions of the system and the conditions under which each function operates. Individually these conditions are often simple but can become quite complex overall. As testers, we need to be able to assure ourselves that every combination that may occur around these conditions has been tested.

A decision table lists all the input conditions that can occur and all the actions that can arise from them.

State Transition Testing

State transition testing is concerned with system outputs which are triggered by changes to the input conditions (changes of state)

Use Case Testing

Use cases offer a way to specify functionality as business scenarios or process flows and allow us to capture interactions between actors and the system.

STRUCTURE-BASED OR WHITE-BOX TECHNIQUES

Structure-based testing techniques are used to explore program structures, they involve generating test cases from code so the who ever is testing needs to be able to read and analyze the code. It may be required to run simple test cases on the code to ensure that all is sound before more detailed functional testing begins, or it may be required to interpret test results from developers to ensure that their testing adequately exercises the code.

Flow Charts

We can use flow charts to represent the code structure we will be testing using flow charts. Flow charts contain two symbols

Diamonds - Represent decisions

Rectangles - Represent sequential statements

Control Flow Graphs

These represent the decision points and the flow of control within a segment of code and are produced by only looking at statements which affect the flow of control.

The graph is made up two symbols

Nodes - Represent any point where the flow of control can be modified

Edges - Is a line connecting any two nodes within a collection of nodes (known as region)

Statement Testing and Coverage

Statement testing is testing aimed at exercising programming statements. If we aim to test every executable statement, this is called 100% statement coverage, if only half the statements are exercised this is called 50% statement coverage and so on.

The reason behind measuring statement coverage is that is a basic measure which ensures testing has been relatively thorough

Decision Testing and Coverage

Decision testing aims to ensure that decisions in a program are adequately exercised when testing decisions we need to test the outcomes for when the decision is both true and false.

Like statement testing, decision testing also has an associated coverage measure and we would normally aim to achieve 100% coverage. This is measured by counting the number of decisions exercised divided by the total number of decisions in a given program and is usually expressed as a percentage.

Other Structure Based Techniques

There are more sophisticated techniques available which can provide increasing complete code coverage. These are essential in some types of applications, such as safety critical apps where it would be vital to know that nothing unacceptable happens at any point where the code is executed.

Coverage is also applicable to other types and levels of structure such as integration level where it may be useful to know what percentage of modules or interface have been executed by a test suite

EXPERIENCE-BASED TECHNIQUES

Experience-based techniques are those used when there is no adequate spec to work from or when there is no time to run the full structured set of tests.

Techniques can range from simple ad hoc testing or error guessing through to t more sophisticated techniques such as exploratory testing. but all tap into the knowledge and experience of the tester rather than systematically exploring a system against a written specification.

Error Guessing

Error guessing is simply the tester using the tester intuition and experience to locate potential errors. When this technique is applied after systematic techniques, error guessing can add another value in identifying and exercising test cases that target known or suspected weaknesses.

The main drawback of error guessing is it's varying levels of effectiveness, depending on the experience of the tester deploying the tests. It can be better used when several tester or/and user contribute to constructing a list of possible errors.

Exploratory Testing

Exploratory testing is a technique that combines the experience of testers with a structured approach to testing where specifications are either missing or inadequate and where there is severe time for pressure. It uses concurrent test design, test execution, test logging and learning when time boxes and is structured test charter containing test objectives.

SYSTEMATIC AND EXPERIENCE-BASED TECHNIQUES

There are some simple rules of thumb when deciding which technique is best.

Always make functional testing the first priority. It may be necessary to test early code products using structural techniques, but we only really learn about the quality of software when we can see what it does.

Then, when basic functional testing is complete, that is a good time to think about test coverage. Have you exercised all functions, requirements, and code? Measures defined at the beginning as exit criteria can now come into play if coverage is inadequate extra test may be needed.

Use structural methods to supplement functional methods where possible. Even if functional coverage is adequate, it will usually be worth checking statement and decision coverage to ensure that enough of the code has been exercised during testing.

Once systematic testing is complete there is an opportunity to use experience-based techniques to ensure that all the most important and most errror-prone areas of the software have been exercised. In some circumstances, such as poor specifications or time pressure, experience-based testing may be the only viable option.

CHOOSING TEST TECHNIQUES

The decisions of which test technique to use is not a simple one. There are many factors to bear in mind, some of which are listed in the box.

Key Selection Factors

Type of system

Regulatory standards

Customer or contractual requirements

Level of risk

Type of risk

Test objectives

Documentation available

Knowledge of the testers

Time and budget

Development life cycle

Use case models

Experience of type of defects found

One or more of these factors may be important on any given occasion.

Página 5

TEST MANAGEMENT - CHAPTER FIVE

RISK AND TESTING

Risk - A factor that could result in future negative consequences, usually expressed as impact and likelihood.

Level of risk - Probability of the risk occurring V's impact if it did happen

Project Risks

Whilst managing the testing project a test leader will use project risks to manage the capability to deliver -

Project Risks Can Include:

Supplier Issues:

Failure of a third party to deliver on time or at all.

Contractual issues, such as meeting acceptance criteria.

Organizational factors:

Skill and staff shortages

Personal and training issues

Political issues, such as a change of management or restructuring that will affect the project resources.

Problems that stop testers communicating their needs and test results

Failure to follow up on low-level testing and reviews

Lack of appreciation of the benefits of testing.

Specialist issues:

Problems in defining the right requirements

The extent that requirements can be met given existing project constraints

The quality of the design, development and test teams

The test manager will look at each of the potential risks and give them a score based on

The likely hood of the risk occurring (Probability)

The impact would have on the project if it did occur (Impact)

A score from 1 - 5 will be given for each with 5 being low risk and 1 being high risk. The score of both will then be added together, the lower the overall score the more serious the risk is to be considered. When this occurs actions should be identified in order to help reduce the risk occurring and reducing the impact if the risk did occur.

Project risks identified during test planning should be documented in the IEEE 829 test plan.

Product Risks

Examples of product risks are:

Error-prone software delivered

Poor requirements leading to badly defined and built software

Potential that a defect in the software/hardware could cause harm to an individual or company

Poor software quality characteristics (e.g. functionality, security, reliability, usability, performance) leading to poor user feedback.

The software does not meet the requirements and delivers functionality that was not requested

TEST ORGANISATION

Test organisation and independence

All humans make errors but's it's more difficult for the author of those errors to see them. In terms of software development, when a developer delivers code, they believe it to be complete and correct, however, when testers receive the code they believe it to contain defects and seek to expose those defects through testing.

Independent testing is important as it is generally harder for authors to see their own mistakes but is easier for other to spot them. There are several levels of independence.

Levels of independence from Low to High

The author/developer

Testers within the same development team

Independent test team within the organization

Independent test team provided by the operational business units

Specialist test such as usability testers, security testers, performance testers

Outsourced test teams or testers.

Independence also comes at a price. The greater the level of independence, the greater the level of bugs being raised through unfamiliarity.

In the real world, organizations and teams will vary in regards to roles and the level of testing independence which can be maintained. For software which is considered safety critical, there will be a need for multiple levels of testing by various teams and testers.

It's important that a tester is able to switch to an independent view of the software and not fall into the same mind set as developers "who own the product". This can be achieved by not assuming anything and ignore the view of "That was how it was developed to work".

Benefits

The tester sees other and different defect to the author

The tester is unbiased

The tester can see what has been built rather than what the developer thought had been built

The tester makes no assumptions regarding quality

Drawbacks

Isolation from the development team, which will mean the tester will be totally dependent on the test basis to understand what needs to be tested.

The tester may be seen as the bottleneck, as independent test execution is usually carried out at the last stage and is affected by any delays earlier in the process

Developers lose a sense of responsibility for quality as it may be assumed that they need not worry about errors because the testers will find them.

The fully independent view sets developers and testers on opposite sides of the fence. This can be a hindrance to communication and common understanding and effective working.

Tasks of a test leader and tester

Test tasks are usually carried out by people who make testing a career, however, test tasks may also be carried out by non-tester such as a project manager, developer and business and domain experts.

The two main testing roles (as covered in the book) are tester and test lead. These roles can be undertaken by anyone who possesses the required skills.

The tasks are undertaken by a test leader align very closely with those undertaken by a project manager and align closely with standard approaches to project manament. In this context, a test leader is anyone who leads a team of testers. Other names for a test leader are Test program manager, test manager, test team leader and test coordinator.

Typical test leader tasks include:

Coordinating the development of the test strategy and plan with project managers and others

Writing or reviewing test strategies produced for the project, and test policies produced for the organization

Contributing the testing perspective to other project activities such as development delivery schedules

Planning the development of the required tests

Managing the specification, preparation, implementation, and execution of tests, including the monitoring and control of all the specification execution.

Taking the required action, including adapting the planning, based on test results

TEST PLANNING AND ESTIMATION

Test Planning

Test planning is the most important activity undertaken by a test leader in any project. It ensures that there is initially a list of tasks and milestones in a baseline plan to track progress against, as well as defining the shape and size of the test effort.

Test planning will be used in green field projects as well as maintenance\support projects.

The main project used in test planning is often called a master test plan or a project test plan. The test plan is normally produced during the early stages of a project and will provide sufficient information to enable a test project to be established.

The details of the test-level activities are documented within test level plans, e.g the system test plan. These documents will contain the detailed activities and estimates for the relevant test levels.

A useful aid to help remember the 16 sections the IEEE 829 test plan is the acronym "SPACEDIRT", each letter mapping to one or several sections of the test plan

S - Scope (Including test items, features o be tested and features not to be tested)

P - People (Including responsibilities, staff and training approvals)

A - Approach

C - Criteria (Including item, pass/fail criteria and suspension and resumption requirements)

E - Environment needs

D - Deliverables (test)

I - Identifier and introduction (test plan)

R - Risks and contingencies

T - Testing tasks and schedule

Test Planning Activities

During test planning various activities have to be undertaken by those working on the plan:

Putting together the overall approach of testing, ensuring that test levels and entry and exit criteria are defined

Liaising with the project manager and making sure that the testing activities have been included within the software life-cycle activities such as:

Design - The development of the software design

Development - The building of the code

Implementation - The activities surrounding implementation into a live environment

Working with the project to decide what needs to be tested, what roles are involved and who will perform the test activities, planning when and how the test activities should be done, deciding how the test results will be evaluated and defining when to stop testing (exit criteria)

Finding and assigning resources to the different tasks that have been defined

Deciding what the documentation for the test project will be, eg which plans, how the test cases will be documented etc.

Defining the management information, including the metrics required and putting in place the processes to monitor and control test preparation and execution, defect resolution and risk issues.

Ensuring that the test documentation generates repeatable test assets eg test cases.

Exit Criteria

Exit criteria are used to determine when a given test activity has been completed or when it should stop. Exit criteria can be defined for all of the test activities, such as planning, specification, and execution as a whole or to a specific test.

Exit criteria should be included in the relevant test plans.

Some typical exit criteria might be:

All tests planned have been run

A certain level of requirements coverage has been achieved

No high-risk areas have been fully tested, with only minor residual risks left outstanding

Cost - when the budget been spent

The schedule has been achieved, eg the release date has been reached and the product has to go live This was the case with the Millennium testing (it had to be completed before midnight on 31 December 1999) and is often the case with government legislation.

Exit criteria should have been agreed as early as possible in the life cycle, however, they can be and often are subject to controlled change as the detail of the project becomes better understood and therefore the ability to meet the criteria is better understood by those responsible for delivery.

Página 6

Check of Understanding

CHAPTER ONE

Describe the interaction between errors, defects, and failures:

Errors lead to defects which the lead to failures

Software failures can cause losses. Give three consequences of software failures:

Loss of money

loss of reputation

Loss of life

What are the vertices of the "triangle of resources":

--------------------------------------------------------

Time

Features

Money Quality

---------------------------------------------------------

Describe static testing and dynamic testing:

Static testing is testing areas of the dev cycle such as testing the requirement specification for errors and ambiguities.

Dynamic testing is performing testing on the application with using test data.

Debugging is when a developer will try to find the cause of an error and apply a fix

What other elements apart from test execution are included in testing?

Test Planning

Documenting Test Results

Why are "Zero defects" an insufficient guide to software quality?

Finding no defects during test execution doesn't mean that there are no defects in the application, it only means that there were no defects found in that execution which could potentially represent an insufficient test plan.

Give three reasons why defect clustering may exist?

System Complexity

Volatile Code

Effects fo change upon change

Developer Experience

Developer Inexperience

Briefly, justify the idea of early testing

By finding defects earlier in the SDLC, costs are reduced as it is far more expensive to find these defects later in the process.

What are the stages in the fundamental test process?

Test Planning & Control

Test Analysis & Design

Test Implementation & Execution

Evaluating Exit Criteria Reporting

Test Closure

Briefly, compare regression testing and confirmation testing

Confirmation testing is testing around an area of work where a bug has been raised and fixed. Its purpose is to confirm that the error where the error was found is now working as expected.

Regression testing is testing around the other areas where a bug was raised to confirm that while fix hasn't caused issue in these other areas

When should the expected outcome of a test be defined?

The expected outcome should be defined in during the analysis and design and used to check that actual outcome matches the expected outcome

When testing software who has the highest level of independence?

Testers who are not a part of the same company doing the development such as a consultancy firm.

Contrast the advantages and disadvantages of developers testing their own code:

Advantages

Able to apply fixes straight away

No need for extensive error logs

No need to raise bugs, test cases etc

Disadvantages

Developers can be blind to mistakes in the code they have written

Developers will modify their behavior to fit project deadlines and objectives, this can lead to incomplete or inadequate testing

Suggest three ways confrontation can be avoided

Always treat errors and defects as software issues, not mistakes a person has made

Keep focus on delivering a quality product

Communicate effectively

CHAPTER TWO

What is meant by verification?

Verification is ensuring the application matches the requirements defined for it.

What is meant by Validation?

Validation is ensuring the application matches the end-users needs.

Name three work products typically shown in the V-model

Requirements Specification, Functional Specification, Technical Specification

Name three activities typically shown in the V-model

Requirements Testing, Functional Testing, Technical Testing

Identify a benefit of the V-Model

The main benefit is that we have solid sets of specifications to test against which makes testing easier.

Name a drawback fo the V-Model

One drawback is, each stage must be completed before the other is able to start, if the requirements are altered or changed part way through, this may not be caught until near the end when testing starts which could be costly.

Name three activities typically associated with the iterative model

Requirements gathering, Design, Coding.

Identify a significant benefit of the iterative model

Being able to test after each iteration offers the benefit of being able to capture issues and fix them in smaller time scales which reduces costs.

List three challenges of an iterative model.

No set and definitive requirements to test against.

Requirements can be changed or adjusted without being documented which could cause confusion.

More regression testing will be required to ensure all parts of the application work following each iteration.

List three types of iterative development.

Agile development, Rapid Application Development, Prototyping.

In the V-Model, which document would be used as the test basis for unit testing?

The technical specification.

Describe three typical integration testing strategies

Big Bang - Testing all components at once (not advisable)

Top-Down The units which will call other are tested first. If a unit to be called has not yet been developed stubs are created.

Bottom-Up The units which receive calls are tested first. If a calling unit has not yet been tested drivers are created.

Identify why stubs and drivers are usually used

As above.

In the V-Model, which document is used as the test basis for system testing,

The Functional Specification

Compare a functional requirement with a non-functional requirement

Functional Requirement is an operation the system must perform such as the ability to add items to a basket

Non-Functional Requirements are generic operations which can usually be applied to most applications such as a system responding to inputs within a reasonable time manner

List Three non-functional requirements

Performance

Maintainability

Usability

What is the purpose of acceptance testing?

The purpose of acceptance testing is to ensure the application meets the users expected requirements and expectations

In the V-model, what is the test basis for acceptance testing?

The Requirments Specification

Identify three types of acceptance testing.

Operational

User

Contract

CHAPTER THREE

Identify three benefits of reviews

Development productivity can be improved and timescales reduced

Improved communication

Testing costs reduced

What happens during the planning phase of a review?

Selecting personnel

Allocating Roles

Defining the entry and exit criteria

Selecting parts of documents to be reviewed

Who manages the review process?

The Manager

Compare the differences between a walk through and an investigation

Name three characteristics of a walk through

Identify at least five success factors for a review

----

What is static analysis looking for?

Looking for defects in the source code and software models

Name the five benefits of static analysis

Earlier detection of defects prior to test execution

Early warning about suspicious aspects of code such as complex functionality

Improved maintainability of code and design

What is the name of the type of defect which relates to password checks?

Security Vulnerability

What defines the process of test execution?

Briefly, compare a test case and a test condition

The test condition is defining what it is we could expect to see based on the test case.

The test case is what will be needed to test the defined

Which document identifies the sequence in which test is executed?

Describe the purpose of a test coverage measures?

CHAPTER FOUR

What is the relationship between a partition and a boundary?

We can use the higher and lower boundaries of an input to partition it's valid or invalid values and use these as or test conditions which we will test against.

Explain what is meant by "as close as possible to a boundary"?

If we have a boundary with a lower of 1 and a higher value of 100, to test the valid conditions which are "as close as possible to the boundary", we would select 3 valid values which would be 99,1001 and 2.

For the invalid value, we do the same using the closest invalid values of -1, 0, 101, 102.

What is the main use of ST for tester

To quickly analysis the various states and events and use it to write up test cases

Name three components of a state transition diagram

State, Event & Transition

How are negative tests identified from an ST

Using a null value to indicate that no transition will take place on that transition.

What is meant by the term "invalid transition"?

This is when the event it self-has no effect and is marked as Null (N)

What is the purpose of a use case?

Use cases allow us to determine the different types of users who will interact with a system and identify test cases based on the user actions.

What is the relationship between a use case and a test case?

A use case helps us identify specific actions to use for a test case.

What is meant by the term executable statement?

Executable refers to code which instructs to the computer to perform some type of action.

Describe the two forms of looping structure so far explained

Do While

This is a type of iterations that says while a condition is true keeping looping otherwise stop

Repeat

This is a type of iteration which behaves similarly to the do while but the condition is at the end meaning the statements inside the loop will always be executed at least once.

What is a selection structure?

A selection structure checks if a condition is true or false using and if, else statement.

How many different paths are there through a selection structure

At least one path but there could be much more

What is the difference between a flow chart and a control flow graph?

A flow chart is used to represent the structure of a program, which captures statements and decisions.

A control flow graph is used to represent only the decisions of a code segment.

Name the three fundamental program structures that can be found in programs?

Sequence

Selection

Iteration

Briefly explain what is meant by an edge, a node and a region in a control flow graph.

Node - Represents any point where the code can be modified

Edge - A line which connects any two nodes

Region - This is the closed region of a collection of nodes and edges

What is the purpose of decision testing?

To determine how many test cases will be required to ensure a thorough enough test has been carried out

How many test cases are needed to exercise a single decision

It will depend on how many statements are then executed from that decision.

How many test cases are needed to exercise a loop structure

It will depend on how many possible decisions there are in the loop structure.

CHAPTER FIVE

What are the two types of risks that have to be considered in testing?

Project risks and product risks.

Compare and contrast these two risk types

Project risks are risks which could have an impact on the delivery of the project it self

Product risks are risks which have an impact on the actual product being built

How early in the life cycle can risk impacting the testing approach?

From the very start

What does MISRA determine when the level of risk is understood

It determines the level of test coverage which should be undertaken

Why is independent testing more effective at finding errors than simply allowing the developer and author to test their own product.

Becuase authors tend not to see mistakes in their own work or something they have ownership in.

Name three benefits of independence.

The tester is unbiased

The tester will find other/different defects to the author

The tester will make no assumptions regarding quality

Which organization provides the lowest level of independence and which provides the highest.

Small organizations where the team take on several roles

Organization which is developing an application which is safety critical

Página 7

EXAMPLE EXAMINATION QUESTION WITH ANSWERS

The Fundamentals of Testing

E1. K1 Question - Which of the following is correct: Debugging is:

C. Identifying the cause of a defect, repairing the code and checking the fix is correct.

E2. K2 Question - Which of the following are aids to good communication and which hinder it

Answer is C

E3 K1 Question - Which option is part of the implementation and execution area of the fundamental test process

B Comparing actual and expected results

E4 K1 Question - The five parts of the fundamental test process have a broad chronological order. Which of the options give three different parts in the correct order

Answer is B

E5 K2 Question - Which pair of definitions is correct?

D. Regression testing is checking that there are no additional problems in previously tested software, confirmation test is demonstrating that the reported defect has been fixed.

E6 K1 Question - Which statement is most true

A. Different testing is needed depending upon the application.

E7 K1 Question - When is testing complete

D. When every data combination has been exercised successfully

E8. K1 Question - Which list of levels of tester independence is in the correct order. starting with the most independent first

Answer is C

E9. K2 Question - The following statements relate to activities that are part of the fundamental test process

Answer is B

Score 8/9 - One Incorrect answer for E7, answer should be B

Life Cycles

E1. K1 Question - Which of the following is usually the test basis for integration testing?

C. Technical specification

E2. K2 Question - A top-down development strategy affects which level of testing most?

B. Integration testing

E3. K2 Question - Which of the following is a non-functional requirement?

D. The system will allow up to 100 users to log in at the same time

E4. K1 Question - Which of the following are examples of iterative development models?

Answer is C

E5. K2 Question - Which of the following statements is true?

Answer is D

E6. K1 Question - Which of the following is not true of regression testing?

A. It can be carried out at each stage of the life cycle

Score 6/6 -

Static Testing

E1 K1 Question - Which of the following is most likely to be a benefit of using static techniques?

B. Productivity improvements in the development process

E2. K2 Question - Which of the following has the typical formal review activities in the correct sequence?

C. Planning, Kick-off, Individual Prep, Review Meeting

E3. K2 Question - Which of the following statements is true?

D.

E4. K1 Question - Which of the following are static techniques?

A. Walkthrough

E5. k1 Question - Which one of the following roles is typically used in a review?

B. Author

E6. K2 Question -Which of the following is most likely to be performed by developers

D. Static analysis of a software model.

Score 6/6

Test Design Techniques

E1. K1 Question - Which of the following scribes structure-based (White box) test case design techniques.

C. Test cases are derived systematically from the delivered code

E2. K1 Question - Which of the following is a structure-based (White box) technique?

B. State transition testing - Incorrect, should be C

E3. K3 Questions

A washing machine has three temperature bands for different kinds of fabrics: fragile fabrics are washed at temperatures between15 and 30 degree Celsius; normal fabrics are washed at temperatures between 31 and 60 degrees Celsus; heavily soiled and tough fabrics are washed at temps between 61 and 100 C. Which of the following contains only values that are in different equivalence partitions?

C. 25, 45, 75

E4. K3 Questions - Consider the following pseudo code:

B. 2

E5. K1 Question - What is the purpose of use case testing?

A. To identify defects in process flows related to tical use of the system.

E6. K2 Question - Which of the following are the most important factors to be taken into account when selecting test techniques

B.

Score 5/6 One Incorrect answer for E2, answer should be C

Página 8

Self-Assesment Questions

CHAPTER ONE

A bug defect is:

C. The result of an Error or Mistake

The effect of testing is to:

B. Give an Indication of the software quality

What is confirmation testing:

D. Running previously failed test against new software/data/documents to see if the problem is solved.

CHAPTER TWO

Which of the following is true about the V-Model?

D. It enables test planning to start as early as possible

Which of the following is true of iterative development?

B. It involves the users in the testing throughout

Which of the following is in the correct order (typically)?

A. Unit Testing, System Testing, Acceptance Testing, Maintenance Testing

Página 9

Why Software Fails

Error - > Defect - > Failure

Incorrect Software Can Harm

People

Companies

The Environment

Software Failure Can Lead To

Loss of money

Loss of time

Loss of business reputation

Injury

Death

Testing and Risk

Greater risk implies more and better testing

Testing and Quality

There will be a level of quality which will be considered acceptable for a given system. Testing doesn't increase the quality itself but instead finds and reports defects making it possible for them to be removed which contributes to the enhanced quality of the system.

Time

Features

Money Quality

Managing Testing Time

Not everything can be tested so we need to prioritize the first test the most important areas which will also address the most significant risks.

The next important aspect is to set criteria which will give an objective test of whether it is safe to stop testing, known as completion criteria.

Página 10

EXERCISES

Chapter 4 - Test Design Techniques

Exercise 4.2

Charge = £3.95 post for order upto £20

= £4.05 for orders above £20

= order above £40 free postage

Partitions

Valid

Less than £20 - Test using £9.99

£20 - £40 - Test using £25

Greater than £40 - Test using £100

Invalid

Zero

Negative Values (Real Numbers)

Decision Tables - Exercise 4.4

Answer: Eligible for cash payments

State Transition Tables - Exercise 4.5

Answer - b. FEABC

Página 11

UDEMY COURSE NOTES

Causes of Software Defects

Faulty Requirements Definition

Time pressures

Complex Code

Lots of interaction between systems

Coding Errors

Complexity of Infrastructure

Changing Technologies

Non-compliance with standards

Test Levels

Unit Testing

Testing the individual units of code, these would be the user stories in a sprint

Integration Testing

Testing how each of the units works together, individual units will be interacting with other units of code

System Testing

Testing the various scenarios and ways in which the system can be used

Acceptance Testing

Testing the software from a users perspective

Static Techniques and the Test Process

Testing software without executing code

Find causes of failures rather than the failures

Usually used before dynamic techniques

Complementary to dynamic techniques

Manual reviews and automated static analysis

Static Testing Types

Reviews

Static Tools

Specification-based Techniques

Equivalence Partitioning (k3)

Inputs are divided into partitions that are expected to exhibit similar behavior

One value from each partition should be selected

Equivalence partitions can be found for both valid data and invalid data

Boundary value analysis (k3)

Decision table testing (k3)

State transition testing (k3)

The state transition technique is concerned with systems that may exhibit a different response depending on current conditions or previous history

It allows the tester to view the software in terms of its states transitions between states, the inputs or events that trigger state changes (transitions) and the actions which may result from those transitions

State transition tables can be used to record all the relationships between the states and inputs which can highlight the possible transitions that are invalid

Creating state-transition tables often unearths combinations that have not been identified in the requirements. It is highly beneficial to discover these defects before coding begins.

Use Case Testing (k2)

Risk and Testing

Risk can be defined as the chance of an event, hazard, threat or situation occurring and resulting in future negative consequences or a potential problem

A risked-based approach to testing provides proactive activities to reduce the risk levels of product risk, as early as possible in the SDLC.

Risks are used to decide where to start testing and where to test more

Risk Management activities:

Risk Identification

Risk Analysis

Risk Response

Risk Analysis

The level of risk will be determined by the likelihood of an adverse event happening and the impact

LEVEL OF RISK = (Probability of the risk) X (Impact if it happens)

Risk Responses

Avoid

Doing whatever can be done to make the probability and impact of the rise zero

Mitigate

Lowering the probability and impact of the risk

Transfer

If possible move the risk from your being your responsibility to someone else

Accept

Having a plan in place which will handle the risk thus reducing the overall impact of that risk

Project Risks (k2)

Technical Issues

Problems in defining the right requirements

Test environment not ready

Old testing tools

Low quality of test basis

Late test basis

Organisational Factors

Skills, Training

Personal Issues

Political Issues

Improper Attitude

Supplier Issues

Failure of a third party

Contractual; issues

Product Risks (K2)

Failure Prone Software

Software/Hardware could cause harm to an individual or a company

Software that does not perform it's intended functions

Poor Software Characteristics

Poor data integrity and quality

Página 12

Test Fundamentals - UC

Software Testing Causes

Faulty Requirements Defenition

Time Pressure

Complex Code

Many System Interactions

Coding Errors

Complexity of Infrastructure

Changing Technologies

Non-Compliance with Standards

Testing & Quality

Testing ensures that key functional and non-functional requirements are met

Testing measures the quality of software in terms of the number of defects found, the tests run and the systems covered by the tests

Testing doesn't directly increase the quality of the software under test but instead, give us an accurate indication of the level of quality of that software, if few or no defects are found this can increase confidence in the quality of the software.

Understanding the root cause of issues found in projects can help prevent those same issues re-occurring in other projects

Testing is just one component of the overall quality assurance activity

Testing Objectives

Finding Defects

Gaining confidence about the level of quality

Providing information decision making

Preventing Defects

Testing Principles

There are seven testing principles:

Testing shows the presence of defects

Cannot prove that there are no defects

Tests should be designed to find as many defects as possible

Exhaustive testing is impossible

Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases

Instead of exhaustive testing, risk analysis and priorities should be used to focus testing efforts

Early Testing

The earlier defects are spotted the less the impact of cost is

Testing activities shall be started as early as possible in the software or system development life cycle and should be focused on defined objectives.

Defect clustering

A small number of modules usually contain most of the defects discovered during pre-release testing or are responsible for most of the operational failures

Pareto Principle says 80 / 20 (read further about this)

Pesticide Paradox

If the same tests are repeated over and over again, eventually the same set of test cases will no longer find any new defects

Test cases need to be regularly reviewed and revised

Testing is context dependent

Testing is done differently in different contexts

Risk can be a large factor in determining the type of testing that is needed

The absence of errors fallacy

Finding and fixing defects does not help if the system built is unusable and does not fulfill the users' needs and expectations

How much testing is enough? (This is the type of question which could be asked)

The answer to this question is that it really depends on the risk of:

Missing important faults

Incurring failure costs

Releasing untested or under-tested software

Losing credibility and market share

Missing a market window

Over-testing, ineffective testing

"Testing should provide sufficient information to stakeholders to make informed decisions"

Risk can then be used to determine:

What to test first

What to test Most

How thoroughly to test each item

What not to test (this time)

Allocate the time available for testing by prioritizing testing

Fundamental Test Process

The fundamental test process consists of the following activities

Planning & Control

Test planning is the act of defining the objectives of testing and the specification of test activities in order to meet the objectives and mission

Test control is the ongoing activity of comparing actual progress against the plan and reporting the status, including deviations from the plan.

Analysis & Design

Test analysis and design is the activity during which general testing objectives are transformed into tangible test conditions and test cases

Implementation and Execution

Test implementation and execution is the activity where test procedures or scripts are specified by combining the test cases in a particular order and including any other information needed for test execution, the environment is set up and the tests are run

Evaluating Exit Criteria and Reporting

Evaluating exit criteria is the activity where test execution is assessed against the defined objectives

Test Closure

Test closure activities occur at project milestones such as when a software system is released, a test project is completed or cancelled or when a maintenance release has been completed

Levels of independent testing

A certain degree of independence often makes the tester more effective at finding defects and failures

The further away a tester is from the owner for the software under test, the more effective that testing will be, it's difficult for a developer to test their own code and there more likely to be blind to the mistakes in their own work.

IEEE-CS/AM Software Engineering Code of Ethics and Professional Practice

Public

Software engineers (SE's) shall act consistently with the public interest

Client & Employer

SE's shall act in a manner that is in the best interest of their client and employer consistent with the public interest

Product

SE's shall ensure that their products and related modifications meet the highest professional standards possible

Judgment

SE's shall maintain integrity and independence in their professional judgment

Management

SE managers and leaders shall subscribe to an promote an ethical approach to the management of software development and maintenance

Profession

SE's shall advance the integrity and reputation of the profession consistent with public interest in mind

Colleagues

SE's shall be fair to and supportive of their colleagues

Self

SE's shall participate in lifelong learning regarding the practice of their profession and shall promote an ethical approach to the practice of the profession

Página 13

Testing Throughout The Software Life Cycle

Software Development Models:

Guidelines

At every stage, a check should be made in order that the work-product for that stage meets its objectives

Work-product evaluation should take place when the product has been declared complete

If the work-product passes its evaluation (test), we can progress to the next stage in confidence

Fixing any problems at the point of creation is cheaper than fixing them at a later stage

Verification and Validation (VV & T)

Verification

Determine whether the work product satisfies the conditions imposed at the start of that phase

Validation

Determination of the correctness of the work product with respect to the user needs and requirements

Testing

The process of exercising software to verify that it satisfies specified requirements and to detect faults

The V-Model

Early test planning and design should be carried out throughout the software development lifecycle

See image:

Iterative-incremental Development Models

There is a difference between incremental and iterative life cycles (do some reading).

Incremental will tend to have a defined requirement at the start where specific sections of the overall project will be completed and added on each iteration.

Iterative development will typically have no solid requirement, to begin with, but the business/user will provide feedback at the end of each iteration which will help mould the project requirements.

It is also possible that many projects will utilize a mix of both Incremental and iterative development

Prototyping, Rapid Application Development, Rational Unified Process (RUP) and Agile are all form of iterative development life cycles

In any life cycle model, there are several characteristics of good testing

For every development activity, there is a corresponding testing activity

Each test level has test objectives specific to that level

The analysis and design of tests for any test level should begin during the corresponding development activity

Testers should be involved in reviewing documents as soon as drafts are available in SDLC

Change Control System - Further Reading here https://www.guru99.com/change-control-business-analyst.html

Testers involved in Iterative models need to have some sort of change control system put in place. This allows testers to keep an eye on software changes as they happen

Test Levels

Acceptance Testing

Acceptance testing is often the responsibility of the customers or users of a system

The goal of acceptance testing is to establish confidence in the system

Finding defects is not the main focus

Acceptance testing may occur at various times in the life cycle for example:

A COTS software product (Commercial Of The Shelve)

A component

Systems Testing

System testing is concerned with the behaviour of a whole system/product

The test environment should correspond to the final target or production environment as much as possible in order to minimize the risk of environment-specific failures not being found in testing.

Typically carried out by an independent test group

Integration Testing

More than one (tested) component

Testing interface and communication between components

Done by designers, analysts, or specialist integration testers.

Unit Testing

Lowest level of testing

Also known as component, module or program testing

Usually performed by the developer/programmers and often are not logged or recorded but instead fixed instantaneously

Unit testing is intended to ensure that the code written for the unit meets it's specification, prior to its integration with other units

Usually tested in isolation using stubs and drivers.

Stubs and Drivers

Typically when a developer is working on a new user story there will be areas of functionality which will need to be tested but the other segments of code which interacts with the written unit have not yet been written.

In order to get around this developers will ofter create whats know as stubs or drivers, which small bits of functional code which mock the missing parts.

A stub is a segment of code which will be called by the unit,

A driver is a segment of code which will make calls to the unit

Integration Testing

Test basis includes

Software and System Design

System architecture

Workflows

Use-cases

Test objects include

Sub-system database implementation

Infrastructure

Interfaces

Integration Types

Big bang integration

Incremental integration

Top-down integration

Bottom-up integration

Functional integration

It's possible for there to be different levels of integration testing

Unit Integration Testing

Tests the interactions between software units and is done after unit testing

Usually done by developers

Systems Integration Testing

Tests the interactions between different systems and may be done after system testing

Usually done by testers

Testers should understand the architecture and influence the integration planning

System Testing

Test basis includes

System and software requirement specification

Use Cases

Functional specification

Test objects include

System, user and operation manuals

System configuration

Configuration data

Acceptance Testing

Test basis includes

User requirements

System requirements

Use cases

Business processes

Risk analysis reports

Test objects include

Business processes on fully integrated system

Operational and maintenance processes

User procedures

Forms

Reports

Configuration data

Acceptance Testing Types

User Acceptance Testing

Typically verifies the fitness for use of the system by business users

Customer (user) should perform or be closely involved

Customer can perform any test they wish, usually based on their business processes

Can happen at any time during the project

Before the final user sign-off

Operational Acceptance Testing

The acceptance the system by the system administrators

Testing of backup/restore

Disaster recovery

User management

Contract and Regulation Acceptance

Testing is against the contract and any documented agreed changes

Testing is performed against any regulations that must be adhered to, such as government, legal or safety regulations.

Alpha & Beta Testing

Alpha testing is performed at the developing organization's site but not by the developing team

Beta testing, or field-testing, is performed by potential customers at their own locations

TEST TYPES

Functional Testing

The functions are "what" the system does

May be described in word products such as a requirements specification, use cases, or a functional specification.

Types of functional testing include

Security Testing

Investigate the functions relating to detection of threats, such as viruses from malicious outsiders

Interoperability Testing

Evaluates the capability of the software product to interact with one or more specified components or systems

Non-Functional Testing

Is the testing of "How" the system works

Non-functional testing includes but is not limited to

performance testing

How many users can connect to the system and how will that affect the performance of the software

load testing

How will the system perform under a heavy number of concurrent users, performing transactions over a certain period of time? How can the system handle heavy load volumes?

stress testing

How will the system perform extreme and intense loads, how does it recover when going back to normal, do KPI's, throughput and response times remain the same as before. Load testing can also expose memory leaks, slowness, security issues and data corruption.

usability testing

Is the system easy to use

maintainability testing

Is the system easy to maintain if we need to fix a defect

reliability testing

Is the system reliable or does it crash eventually

portability testing

Is the system easy to port from one platform to another

Structural Testing (AKA Wide box Testing)

Coverage is the extent that a structure has been exercised by testing, expressed as a percentage of the items being covered

If coverage is not 100%, then more tests may be designed to test those items that were missed to increase coverage

Testing Related to Changes

After a defect is detected and fixed, the software should be retested to confirm that the original defect has been successfully removed. This is called CONFIRMATION testing or Re-Testing.

REGRESSION testing is the repeated testing of an already tested program, after modification, to discover any defects introduced or uncovered as a result of the changes

Maintenance Testing

Testing which takes place on a system which is in operation in the live environment is called maintenance testing

Triggers of Maintenance Testing

Modification

Planned enhancement changes

Corrective and emergency changes

Changes in environment

Migration

From one platform to another

Conversion testing

Testing of data migration

Archiving the data

Includes Extensive Regression Testing

Things to consider

What could this change have an impact on

How important a fault in the impacted area

Test what has been affected, but how much

Most important affected areas

Areas most likely to be affected

Whole system

Página 14

Static Techniques and the Test Process

KiStatic Techniques allow us to

Test software without executing the code

Find causes of failures rather than the failures

Usually used before dynamic techniques

Complementary to dynamic techniques

Manual reviews and automated static analysis

STATIC TESTING TYPES

Reviews

A review is a systematic examination of a document by one or more people with the aim of finding and removing errors

Reviews can be used to test anything that is written or typed; this can include documents such as requirement specifications, system designs, code, test plans and test cases

Static Tools

BENEFITS OF REVIEWS

Early feedback on quality issues can be established

Development productivity improvement

Improved team communication and knowledge transfer

Lifetime cost reductions

Reduced fault levels

Reduced development timescales

Reduced testing time and cost

TYPICAL DEFECTS FOUND IN REVIEWS

Requirement defects

Design Defects

Insufficient Maintainability

Incorrect interface specifications

Deviations from standards

REVIEW PROCESS

Activites of Formal Review

Planning

Plan the review activities

Selecting the personnel

Allocating roles

Defining entry & exit criteria

Entry - Does it contain a minimum set of specifications and categories

Exit - What should be met at a minimum, before exit ie. identified bugs in requirement fixed

Kick-off

Distributing documents

Explaining the objectives and process

Can be run through a meeting or just sending an email

Checking entry criteria

To ensure that the reviewers time is not wasted on a document that is not ready for review

Individual preparation

Reading documents before meetings etc should be time-boxed 2-4 hours

Noting incidents

Should be time-boxed

Logging potential defects suggestions and questions

Should watch for the checking rate which is the number of pages checked per hour

Review meeting

Discussing defects

More defects will be found by discussion

Just recommendations on how to fix a defect

Every defect and it's severity should be logged. The participant who identifies the defect proposes the severity

The approach of the meeting depends upon:

Time available

Review objectives

Type of review

Examine

Analyze the recording of the review output

Rework

Checking the logged defected and answer questions

Fixing defects

Update the status of the defects

Follow-up

Assure defects fixed

Collect metrics, such as time spent on review and number of defects found

Checking exit criteria

To officially close the review process for that document

Formal Review Roles and Responsibilities

Author

Leader / Moderator

Reviewers / Inspecters

Managers

Scribe / Recorder

Factors Effecting Review Formality

The maturity of the development process

Legal or regulatory requirements

The need for an audit trail

Review Types

Informal (No Process, Quick Benefit)

Informal process

May take the form of pair programming

Results usually not documented

Varies in usefulness depending on the reviewers

An inexpensive way to find defects and get some benefit

Walkthrough (Led by the author, Learning gaining understanding & optional activities)

Meeting led by author

Optional process

May vary in practice from quite informal to very formal

Main purposes: Learning, gaining understanding, gathering information & finding defects

Technical Review (Led by trained moderator, Discussing, Taking decisions and Evaluating alternatives, Most activities are executed, )

Ideally led by a trained moderator

Mostly follow process

May vary in practice from quite informal to very informal

Main purposes: discussing, making decisions & evaluating alternatives

Inspection (Led by trained moderator, Finding Defects, All activities are executed )

Led by trained moderator

Formal process

Main purpose: finding defects

Success Factors

Each review has clear predefined objectives

The right people for the review objectives are involved

The review is conducted in an atmosphere of trust

Will not be used for the evaluation of the participants

Defects found are welcomed and expressed objectively

People issues and psychological aspects are dealt with

Checklists or roles are used if appropriate

Traning is given in review techniques

Management supports a good review process

Testers are welcome as reviewers

Emphasis on learning and process improvement

Static Analysis Tools

Find defects rather than failures

Analyze program code and software models

Typically used by developers

Ideally performed before the types of formal review

Types of defects found would be hard to find during test execution

Can also be used to calculate code metrics

Value of Static Analysis Tools

Early detection of defects prior to test execution

Early warning about suspicious code or design

Identification of hard to find defects

Detecting inconsistencies in software models

Improved maintainability of code and design

Prevention of defects

Typical Defects Discovered By Static Analysis Tools

Página 15

Fundamental Test Process

Type in The Fundamental test process consists of the following activities

Planning & Control

Analysis & Design

Implementation and Execution

Evaluating Exit Criteria and Reporting

Test Closure

Analysis & Design

The analysis is to: Decide on a test condition, a test condition being a work item or event of a component or system that could be verified by one or more test cases, e.g. A function, transaction, feature, quality attribute or structural element.

Each test condition will be located in the Test Design specification document

The test condition will need to be prioritized

The design is to: Design a test case that will verify the test condition(s) or test objective(s)

Test Cases

A set of input values, preconditions and expected result and postconditions developed for a particular objective or test condition.

It's possible for a test case to cover more than one test condition

All test cases should be located in the Test Design specification document

Each test case should be prioritized

Implementation and Execution

Implementation involves writing a test procedure to execute the test and writing out a test execution schedule.

The test procedure outlines each of the steps to be taken in order to carry out the execution of test cases or cases against the test conditions

Test procedures should be located in the test procedure specification document, and need to be prioritized

Each procedure should be prioritized so that testing is carried out in an effective manner saving both time and cost

Execution Involves running through each of the test procedures, logging the results and reporting any discrepancies to the developers, project manager, and other interested parties.

-- EXAMPLE -------------------------------------------

Test Condition

REQ 1.A - The age of the employee shall be in the range from 20 to 50, whole numbers only

Test Case

Test Range 20 - 50

10

40

60

Test Whole Numbers

40

1.5

Test Numbers

40

xyz

$%^

Test Procedure One

Launch the "ExpertWave" application

Select the "New Employee" menu item from the "Employee" menu, the "New Employee" dialog should be displayed

Check that the cursor blinking inside the "age" field

Type in '40' and click the 'OK' button

Check that the message "Thank you" is displayed

Test Procedure Two

Launch the "ExpertWave application"

Type Ctrl-N, the "New Employee" dialog should be displayed

Check that the cursor blinking inside the age field

Type in '30.4' and press the Enter key

Check that the Alert "Age should be whole numbers only" is displayed

-- EXAMPLE END --------------------------------------------------

CATEGORIES OF TEST DESIGN TECHNIQUES

What is a testing technique?

The purpose of a test design technique is to identify test conditions, test cases, and test data

Effective Testing

Finding more faults

Focus attention on specific types of fault

Know you're testing the right thing

Efficient Testing

Find faults with less effort

Avoid duplication

Systematic techniques are measurable

Categories (here but are many more)

Specification-based test design techniques (Black Box)

Test cases derived systematically from Models, either formal informal

Structure-based test design techniques (White Box)

Test cases derived from information about how the software is constructed (e.g, Code or Database)

Experience-based test design techniques (User Testing)

The knowledge and experience of people are used to derive the test cases

Specification-based testing (Black Box)

Equivalence partitioning

Inputs are divided into partitions that are expected to exhibit similar behavior

One value from each partition should be selected

Equivalence partitions can be found for both valid data and invalid data

Boundary value analysis

boundaries are an area where testing is likely to yield defects

Tests can be designed to cover both valid and invalid boundary values. When designing test cases, a test for each boundary value is chosen

This technique is often considered as an extension of equivalence partitioning

Boundary value analysis can be applied at all levels

Decision table testing

is used by creating a table which contains action and condition columns which contain each potential scenario.

State transition testing

The state transition technique is concerned with systems that may exhibit a different response de[ending on current conditions or previous history

It allows the tester to view the software in terms of its states. transitions between stats, the inputs or events that trigger state changes (transitions) and the actions which may result from those transitions

A state table shows the relationship between the states and inputs and can highlight possible transitions that are invalid

Use case testing

A use case describes interactions between actors (users or external systems) and the system, which produces a result of value to a system user or the customer

Test cases derived from use cases are most useful in uncovering defects in the process flows during real-world use of the system

A use case usually has a most likely scenario and alternative path, precondition, postcondition and final state of the system after the use case has been completed

Use cases are very useful for deigning acceptance tests with customer/user participation

Are an excellent basis for system level testing

They also help uncover integration defects caused by the interaction and interference of different components

Designing test cases from use cases may be combined with other specification bases test techniques

Structure-based Testing (White Box)

Is based on the identified structure of the software or the system

Structure-based techniques serve two purposes

Test coverage measurement

Structural test case design

They are often used to assess the amount of testing performed by tests derived from specification-based techniques, i.e to assess coverage.

They are then used design additional test with the aim of increasing the test coverage.

Statement Testing and Coverage

Statement coverage is determined by the number of executable statements covered by (designed or executed) test cases divided by the number of all executable statements in the code under test.

If we test every executable statement we call this full or 100 percent statement coverage.

Statement coverage is the weakest white-box technique

Decision Testing and Coverage

Decision coverage, related to branch testing, is the assessment of the percentage of decision outcomes (e.g, the true and false options of an IF statement) that have been exercised by a test case suite

Decision coverage is determined by the number of all decision outcomes covered by test cases divided by the number of all possible decision outcomes in the code under test.

Decision coverage is stronger than statement coverage; 1005 decision coverage guarantees 100% statement coverage but not vice versa

Other sTructure-based

There are stronger levels of structural coverage beyond decision coverage, for example, condition coverage and multiple condition coverage

Coverage can also be applied at other test levels

Coverage can also be applied to black box testing techniques

Tool support is useful for the structural testing of code

Página 16

Test Strategy and Test Approach

TYPICAL TEST APPROACHES

The test approach is the implementation of the test strategy for a specific project.

A test approach includes all of the decisions made on how testing should be implemented, based on the (test) project goals and objectives, as well as the risk assessment.

It forms the starting point for test planning, selecting the test design techniques and test types to be employed. It should also define the software and test entry and exit criteria.

Analytical

Risk-based strategy

Requirements-based strategy

Model-Based

Stochastic testing

Methodical

Failure-based (including error guessing and fault attacks)

Checklist based

Quality-characteristic based

Process / standard-compliant

IEEE 829 standard

Agile methodologies (e.g. Extreme programming)

Dynamic and heuristic

Error guessing

Exploratory testing

Consultative

Ask the users or developers

Regression-averse

Test automation