Notes
1 - The Fundamentals of Testing
Content of the notes (16)
Page 1
THE FUNDAMENTALS OF TESTING - CHAPTER ONE
WHAT TESTING IS AND WHAT TESTING DOES
Testing makes a rigorous examination of the behaviour of a component or system and reports all defects found for the development team to correct.
Testing then repeats enough tests ensure that defect corrections have been effective.
Static & Dynamic Testing
Static testing is testing where there is no code, such as testing the requirement documentation
Dynamic testing is when we test an application using the app and it's some form of testing data, this can be referred to as test execution.
GENERAL TESTING PRINCIPLES
The purpose of testing is to find defects, testing can only show that one or more defects exist, it can't show that a program is errors.
Early Testing
In terms of cost, it is far cheaper to find errors at the earlier stages of the SLDC than at the later stages
Defect Clustering
About 80% of defects found in an application are found in about 20% of the modules. It is therefore useful if testing activity reflects this by targeting areas where a high number of defects have been found. There may be fewer defects in the other 80% but they should still be tested diligently for bugs.
FUNDAMENTAL TEST PROCESS
The fundamental test process consists of five parts that encompass all aspects of testing.
Planning & Control
Analysis and Design
Implementation & Execution
Evaluating Exit Criteria and Reporting
Test Closure Activities
Planning & Control
Test planning is determining what is going to be tested and how this will be achieved, it is also where the test completion criteria is defined.
Control is what we do when the testing activities don't match up with the plans. It's an ongoing activity where we compare the progress against the plan.
Typical Tasks Would Be
Defining the scope and objectives of testing and identifying risks
Determining the test approach
Detailing what is required to carry out the testing
Implementing test policies and strategies
Scheduling test analysis and design tasks
Scheduling test implementation, execution, and evaluation
Detailing when testing will stop ie the exit criteria
Measuring and analyzing results
Comparing expected and actual progress, test coverage and exit criteria
Making corrections if things go wrong and deciding actions
Typical Test Case Design Techniques
Reviewing requirements, architecture, design, interfaces and other parts which collectively comprise the test basis
Analysing test items, specifications, behavior and structure to identify test conditions and test data required
Designing the tests
Determining whether the requirements and the system are testable
Detailing what the test environment should look like and whether there are any infrastructure and tools required
Test Implementation and Execution
Evaluating exit criteria and reporting
Testing closure activities
Testing closure activities concentrate on making sure that everything is complete and tidied away, such as reports written, defects closed and deferred for another phase of testing
Typical Test Closure Tasks
Ensure all documentation is in order and up to date
Closing down and arching the test environment
Passing over test ware to the maintenance team
Writing down any lessons learned from the testing project.
THE PSYCHOLOGY OF TESTING
The aim is to work together rather than be confrontational, keep the focus on delivering a quality product.
Results should be presented in a non-personal way. The work item may have errors or be incorrect so point this out in a non-personal way.
Attempt to understand how others feel, it is possible to discuss problems and still leave others feeling positive
Confirm understanding after conversations & discussions
Page 2
LIFE CYCLES - CHAPTER TWO
SOFTWARE DEVELOPMENT MODELS
A typical development life cycle for a software product involves
Requirement Specification
Functional Specification
Technical Specification
Program Specification
Coding
Testing
The above steps are usually used in a waterfall model which is known as a linear or sequential model. In this model, all testing is carried out at the end when all coding has been completed where it will either pass or fail. One of the main issues with this model is if certain parts of the application fail, under normal circumstances it would be difficult to separate those parts from the rest of the application and only release the passed areas.
Verification
Checks that the work product meets the requirements set out for it. This helps to ensure that we are building the product in the correct way.
Validation
Changes the focus of work product evaluation to evaluation against the user needs. This means ensuring that the behavior of the work product matches the customer needs as defined for the project. This helps to ensure we are building the correct product as far as the end-users are concerned.
Development Models
There are two types of development model which facilitate early work-product evaluation
V Model
The V-Model is an extension of the waterfall model. The Left-hand side of the model focuses on elaborating the on the initial requirements.
Requirement specification - capturing user needs
Functional specification - definition of functions required to meet user needs
Technical specification - technical design of functions identified in the functional spec
Program specification - detailed design of each module or unit to be built to meet required functionality
The middle of the V-model shows that planning for testing should start with each work product.
The right-hand side of the V-model focuses on testing activities where for each work product a testing activity is identified
Testing against the requirements spec takes place at the acceptance testing stage
Testing against the functional spec takes place at the systems testing stage
Testing against the technical spec take place at the integration testing stage
Testing against the program spec takes place at the unit testing stage.
Each stage must be completed before the next one can be started, this approach software development pushes validation of the system by the user representatives right to the end of the life cycle. If requirements are not captured correctly or are changed, issues may not be uncovered until user testing is carried out. Fixing issues at this stage could be very costly and could lead to the project being canceled.
Iterative Development Models
In this model, the requirements do not need to be fully defined before coding can start. Instead, a working version is built in a series of iterations/stages where each iteration encompasses requirement definition, design, code, and test.
The project will have a timescale and cost with stages which have defined timescales and costs.
Forms of iterative development include prototyping, rapid application development (RAD) and agile software development.
TEST LEVELS
Characteristics of good testing across development life cycle include:
Early test design
Each work-product is tested
Testers are involved in reviewing requirements before they are released.
Typical Levels of testing are
Unit Testing
Integration Testing
System Testing
Acceptance Testing
Unit Testing
Generally, the code will be written in parts known as units or modules. Unit testing is intended to ensure that the code written for the unit meets the specification and that all code can be fully executed. Unit tests are usually performed by a developer and defects found tend not to be documented.
Integration testing
Once all units have been tested and passed they are integrated together. The next stage of testing is to test all of the integrated units. The purpose of integration testing is to expose defects in the interfaces and in the interactions between integrated components or systems.
Before integration testing can be planned, an integration strategy is required which involves making decisions on how the system will be put together before testing.
Big Bang Integration
This is where all units are linked at once, resulting in a complete system. It's difficult to isolate defects and is considered a bad choice as it introduces the risk that issues will be found late in the project resulting in high costs.
Top Down Integration
This is where the system is built in stages starting with components which call other components. Components that call other are usually placed above components which are called.
Each component must be tested when it is built. If a component to be called has not yet been integrated a stub may be used to replace it
Bottom Up Integration
This is the opposite of top-down integration. If top level components which will make calls have not yet been written, then drivers may be used to replace them.
Levels of Integration Testing
There may be more than one level of integration testing
Component integration testing - focuses on the actions between components and is completed after unit testing and is usually performed by developers.
System integration testing - focuses on the interactions between different systems and is completed after individual testing of each system. This type of integration testing is usually performed by testers.
SYSTEM TESTING
Once all components have been tested and are working at a unit & integration level, the next step is to consider the functionality from an end-to-end level.
System testing is required because testing at unit and integration level does not take into account interactions across the whole system and potential errors due to the environment.
System testing should be carried out in an environment replicating the live environment and would typically be carried out by a team which is independent of the development process.
In the V-Model the behavior required for the system is defined in the functional specification, which contains definitions for both functional and non-functional requirements.
Functional Testing focuses on what the system should do. Non-functional system testing looks at aspects which are important but not directly related to what functions the system performs. Non-Functional requirements tend to be generic requirements which can be applied to many different systems, such are responding to inputs in a reasonable time manner. Typically these requirements will consider both normal operations and behavior under exceptional circumstances.
Examples of non-functional requirements
Installability - installation procedures
Interoperability - operation of the application in different environments
Maintainability - ability to introduce changes to the system
Performance - expected normal behavior
Load Handling - Behaviour of the system under increasing load
Stress Handling - Behavior at the upper limits of system capability
Portability - Use on different operating systems
Recovery - Recovery procedures on failure.
Reliability - Ability of the software to perform its required functions over time
Usability - ease with which users can engage with the system
* Note - Security is regarded as functional requirements in the learning syllabus
ACCEPTANCE TESTING
The next step after system testing is acceptance testing. The purpose of acceptance testing is to provide the end user with confidence that the system will function according to their expectations. The key purpose of acceptance testing is to demonstrate system conformance to the customer requirements.
In the V-Model acceptance testing will be carried out using the requirements spec as a basis for testing.
Acceptance testing is usually the responsibility of the customer or users of the system.
Typical forms of acceptance testing include the following:
User Acceptance Testing - Testing by user representatives to check that the system meets their business needs.
Operational Acceptance Testing - Involves checking that processes and procedures are in place to allow the system to be used and maintained such as Backup files, Procedure for disaster recovery, Training for end users, Maintenance Procedures, Security Procedures.
Contract and Regulation Acceptance Testing -
CAT is when the criteria for accepting a system is documented in a contract. Testing will eventually be conducted to check that these criteria have been met before the system is accepted.
R-A-T Regulation acceptance is checking that the system meets regulations and legislative demands as required for systems built in with certain sectors such as government, defense, banking, pharmaceutical.
Alpha & Beta Testing -
Alpha Testing is testing that takes place at the site where the system was developed
Beta Testing is testing which takes place at the customer's site.
TESTING RELATED TO CHANGES
Confirmation testing is re-testing a part of the program where a defect has previously been found to confirm that the found defect has now been removed.
Regression testing is re-testing the unchanged software to ensure no other issues have been introduced during a bug fix. Regression testing should also be carried out following any change to the environment. Regression testing involves creating a set of tests which serve to demonstrate t the system works as expected. These tests are likely to be used many times, so regression testing is a suitable candidate to be for automation.
MAINTENANCE TESTING
Nearly all project will eventually be released to a live environment. During their deployment, there may be various changes required such as:
Additional features added
Migrations to new operating systems/environments
System being retired
New faults being found and fixed
Any changes made to the live system must be tested and ideally, the application should be subject to regression testing. It would likely be unfeasible and costly to regression test all areas, so it must be determined which areas are most likely to be affected by the new changes, this is referred to as impact analysis. Impact analysis can be difficult, as the original requirements may have changed or other issues such as the original developers are no longer available and an overall lack of documentation or out of date documentation exists.
Page 3
STATIC TESTING - CHAPTER THREE
Static Techniques
Static testing is a technique which tests around software without actually executing any code. This typically includes checking requirements and specification documentation.
The first of these is known as review and is typically used to find and remove errors and ambiguities in documents before they are used for development reducing cost and time.
The second is known as static analysis, which enables code to be analyzed for structural defects or systematic programming weaknesses that may lead to defects. This is usually completed using automation tooling.
Reviews
Reviews can be used to test against anything such as requirements, system designs, code and test plans. Reviewing code against development standards can prevent defects appearing in test execution, although as the code has already been written, not all potentials costs will be greatly reduced.
Important Benefits of finding defects early
Development productivity can be improved and time scales reduced.
Testing cost and time will be reduced by removing the main delays in test execution.
Reduction in lifetime costs can be achieved because fewer defects in the software means fewer support costs
Results in improved communication and clarity of requirements
Types of Defects typically found by reviews
Deviations from standards which have either been internally defined or managed by legislation and regulations.
Requirements defects - The design not matching the requirements
Insufficient maintainability - The code being too complex to maintain
Incorrect interface specs
Review Process
The formality of a review process can vary based on the following:
The maturity of the development process, the more mature, the more formal
Legal or regularity requirements
The need for an audit trail. A formal review process ensures that it's possible to trace back through the SDLC.
Reviews can have a variety of objectives with review objective defining the main focus of the review.
Finding defects
Gaining understanding
Generating discussion
Decision making by consensus
All reviews will exhibit the same basic elements of process
The document under review is studied by the reviewers
Reviewers identify issues or problems and inform the author either verbally or in the documentation.
The author decided on any action to take in response to the comments and updates the document accordingly
Phases of a formal review
Planning
Selecting the personnel
Allocating Roles
Defining the entry and exit criteria
Selecting which parts and which documents should be reviewed
Kick-Off
Individual Preparation
Review Meeting
What Time is available
Requirements of the author
Type of review
Rework
Follow-up
Roles and Responsibilities
The role of each reviewer is to look at the documents belonging to them from their assigned perspective. Checklists may be used which are based on that user's perspective such as user, maintainer, tester or operations.
Type of review
A single document may be subject to many different review types. An informal review may be carried out before the document is subject to a technical review or it could also be possible depending on the level of risk that a technical review or inspection may take place efore a walkthrough with a customer.
Levels of formality by review type
Informal
No formal process underpins the review
Maybe documented but is not required
Maybe variations in the skills of the reviewer
The main purpose is to find defects
May be implemented using pair programming (One dev reviews code of another) or reviewed by a technical lead
Walkthrough
Led by author of document and attended by members of authors peer group
Review sessions are open-ended
Technical Review
Inspection
STATIC ANALYSIS BY TOOLS
Static analysis is looking for defect without executing the code once. Using static analysis we can find defects that are hard to during test execution by analyzing the program code e.g instructions to the computer can be in the form of control flow graphs and data flows
The value of static analysis is:
Early detection of defects prior to test execution.
Early warning about suspicious aspects of the code or design
Identification of defects not easily found by dynamic testing
Improved maintainability of code and design
Prevention of defects
Typical defects discovered by static analysis tools include:
Referencing a variable with an undefined value
Inconsistent interface between modules and components
Variables that are never used
Unreachable (dead) code
Programming standard violations
Security vulnerabilities
Syntax Violations
Page 4
TEST DESIGN TECHNIQUES - CHAPTER FOUR
TEST CONDITIONS, TEST CASES, AND TEST PROCEDURES
The design of tests comprises three main steps
Identify test conditions - What we test for
Specify test cases - What is used to test with
Specify test procedures - Steps taken to carry out the test
Test Condition - An item or event of a component or system that could be verified by one or more test cases, e.g. a function, feature, quality attribute or structural elements. A test condition is some characteristic of our software that we can check with a test or a set of tests, it is the condition we would expect to see happen that we test for.
Test Case - A set of input values, execution preconditions, expected results, and execution, post conditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. A test case gets the system to some starting point for the tests, then applies a set of input values that should achieve a given outcome and leaves the system at some end point.
We should define here what state the system is in when we start and what state it is in after the test in order to check it ends up in the correct place.
Test Procedure Specification - A sequence of actions for the execution of a test. A test procedure identifies all the necessary action in sequence to execute a test, these can also be called test scripts or manual test scripts.
Test Execution Schedule - This is used to test the whole system by placing together each of the individual test procedures in a sequence of steps in order of highest priority to be executed. We should also think about ordering in terms of what makes the most sense when testing and group relevant tests together.
THE IDEA OF TEST COVERAGE
Test coverage provides a quantitative assessment of the extent and quality of testing, answering the question of how much testing was done.
There are two important reasons why we need to know about test coverage:
It provides a quantitative measure of the quality of the testing has been done by measuring what has been achieved
It provides a way of measuring how much more testing needs to be done Using quantitative measures we can set targets for test coverage and measure against them.
CATEGORIES OF TEST CASE DESIGN TECHNIQUES
The main techniques to learn here will be
Specification-based / Black-box Techniques - Deriving test cases directly from the specification or the model of a proposed system.
Structure-based / While-box Techniques - Deriving test cases directly from the code written to implement a system.
Experience-based Techniques - Deriving tests cases from the testers experience of similar systems and general experience of testing.
These are not the only categories available
SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES
The source of information on which to base testing is knowing as the "Test Basis", this will typically be a requirement spec of model spec. It the test basis is well defined and adequately structured we can easily identify test condition from which to test against.
If a detailed requirements spec is available it is better as a tester to write test cases using this, as it increases the chance of finding defects and you are testing directly against what was specified. If you build test cases against the already built program it becomes more difficult to identify what is an actual defect or what is a genuine feature enhancement.
If no detailed specification is available, a tester may need to speak with key stakeholders in order to determine the program's expectations.
It's also likely that a specification will contain nonfunctional requirements such as usability, reliability, and performance expectations.
Five Specification Base Techniques
Equivalence partitioning
Boundary value analysis
Decision table testing
State transition testing
Use case testing
Equivalence partitioning
Input Partitions
We can use input partitions to group the test conditions where we expect the program to pass or fail. If we had an input should only expect values -10,000 to 10,000, then the passing values could be split up into three separate partitions
Negative Valid Values - Positive Valid Values - Zero
This then allows us to choose a number to represent each of these groups and typically a number in the middle range would be chosen.
We would then need to consider invalid values and split those into various partitions also, however as with most test there will be a greater range of potential invalid values than valid, because of this we may need to limit our testing to using values which would be the most likely to crop up in a real world situation.
Output portions
If enough is known about an application we may be able to partition other values instead of just input/output values. If say a program handled input values by placing them in queues we could in principle check that the requests end up in the correct queue. In this, we could then create a number of partitions based on the correct queue.
Boundry Value Analysis
Boundry value analysis looks at the min & max valid values for some input. It's important to test and consider these as boundaries are an area where mistakes are typically made for various reasons such as a for loop making one less or more loop than it should have. If a program accepted a number with the range 1 - 100 then
1 would be lower valid boundary
100 would be the higher valid boundary
We would also have non-valid boundaries which in this case would be
0 would be a lower invalid boundary
101 would be a higher invalid boundary
We would then test each end of the boundary so could run tests against
Valid Test 0 1 2
Invalid Test 99 100 101
Decision Table Testing
Decision tables are used to capture all logical decisions. Specifications often contain business rules to define functions of the system and the conditions under which each function operates. Individually these conditions are often simple but can become quite complex overall. As testers, we need to be able to assure ourselves that every combination that may occur around these conditions has been tested.
A decision table lists all the input conditions that can occur and all the actions that can arise from them.
State Transition Testing
State transition testing is concerned with system outputs which are triggered by changes to the input conditions (changes of state)
Use Case Testing
Use cases offer a way to specify functionality as business scenarios or process flows and allow us to capture interactions between actors and the system.
STRUCTURE-BASED OR WHITE-BOX TECHNIQUES
Structure-based testing techniques are used to explore program structures, they involve generating test cases from code so the who ever is testing needs to be able to read and analyze the code. It may be required to run simple test cases on the code to ensure that all is sound before more detailed functional testing begins, or it may be required to interpret test results from developers to ensure that their testing adequately exercises the code.
Flow Charts
We can use flow charts to represent the code structure we will be testing using flow charts. Flow charts contain two symbols
Diamonds - Represent decisions
Rectangles - Represent sequential statements
Control Flow Graphs
These represent the decision points and the flow of control within a segment of code and are produced by only looking at statements which affect the flow of control.
The graph is made up two symbols
Nodes - Represent any point where the flow of control can be modified
Edges - Is a line connecting any two nodes within a collection of nodes (known as region)
Statement Testing and Coverage
Statement testing is testing aimed at exercising programming statements. If we aim to test every executable statement, this is called 100% statement coverage, if only half the statements are exercised this is called 50% statement coverage and so on.
The reason behind measuring statement coverage is that is a basic measure which ensures testing has been relatively thorough
Decision Testing and Coverage
Decision testing aims to ensure that decisions in a program are adequately exercised when testing decisions we need to test the outcomes for when the decision is both true and false.
Like statement testing, decision testing also has an associated coverage measure and we would normally aim to achieve 100% coverage. This is measured by counting the number of decisions exercised divided by the total number of decisions in a given program and is usually expressed as a percentage.
Other Structure Based Techniques
There are more sophisticated techniques available which can provide increasing complete code coverage. These are essential in some types of applications, such as safety critical apps where it would be vital to know that nothing unacceptable happens at any point where the code is executed.
Coverage is also applicable to other types and levels of structure such as integration level where it may be useful to know what percentage of modules or interface have been executed by a test suite
EXPERIENCE-BASED TECHNIQUES
Experience-based techniques are those used when there is no adequate spec to work from or when there is no time to run the full structured set of tests.
Techniques can range from simple ad hoc testing or error guessing through to t more sophisticated techniques such as exploratory testing. but all tap into the knowledge and experience of the tester rather than systematically exploring a system against a written specification.
Error Guessing
Error guessing is simply the tester using the tester intuition and experience to locate potential errors. When this technique is applied after systematic techniques, error guessing can add another value in identifying and exercising test cases that target known or suspected weaknesses.
The main drawback of error guessing is it's varying levels of effectiveness, depending on the experience of the tester deploying the tests. It can be better used when several tester or/and user contribute to constructing a list of possible errors.
Exploratory Testing
Exploratory testing is a technique that combines the experience of testers with a structured approach to testing where specifications are either missing or inadequate and where there is severe time for pressure. It uses concurrent test design, test execution, test logging and learning when time boxes and is structured test charter containing test objectives.
SYSTEMATIC AND EXPERIENCE-BASED TECHNIQUES
There are some simple rules of thumb when deciding which technique is best.
Always make functional testing the first priority. It may be necessary to test early code products using structural techniques, but we only really learn about the quality of software when we can see what it does.
Then, when basic functional testing is complete, that is a good time to think about test coverage. Have you exercised all functions, requirements, and code? Measures defined at the beginning as exit criteria can now come into play if coverage is inadequate extra test may be needed.
Use structural methods to supplement functional methods where possible. Even if functional coverage is adequate, it will usually be worth checking statement and decision coverage to ensure that enough of the code has been exercised during testing.
Once systematic testing is complete there is an opportunity to use experience-based techniques to ensure that all the most important and most errror-prone areas of the software have been exercised. In some circumstances, such as poor specifications or time pressure, experience-based testing may be the only viable option.
CHOOSING TEST TECHNIQUES
The decisions of which test technique to use is not a simple one. There are many factors to bear in mind, some of which are listed in the box.
Key Selection Factors
Type of system
Regulatory standards
Customer or contractual requirements
Level of risk
Type of risk
Test objectives
Documentation available
Knowledge of the testers
Time and budget
Development life cycle
Use case models
Experience of type of defects found
One or more of these factors may be important on any given occasion.
Page 5
TEST MANAGEMENT - CHAPTER FIVE
RISK AND TESTING
Risk - A factor that could result in future negative consequences, usually expressed as impact and likelihood.
Level of risk - Probability of the risk occurring V's impact if it did happen
Project Risks
Whilst managing the testing project a test leader will use project risks to manage the capability to deliver -
Project Risks Can Include:
Supplier Issues:
Failure of a third party to deliver on time or at all.
Contractual issues, such as meeting acceptance criteria.
Organizational factors:
Skill and staff shortages
Personal and training issues
Political issues, such as a change of management or restructuring that will affect the project resources.
Problems that stop testers communicating their needs and test results
Failure to follow up on low-level testing and reviews
Lack of appreciation of the benefits of testing.
Specialist issues:
Problems in defining the right requirements
The extent that requirements can be met given existing project constraints
The quality of the design, development and test teams
The test manager will look at each of the potential risks and give them a score based on
The likely hood of the risk occurring (Probability)
The impact would have on the project if it did occur (Impact)
A score from 1 - 5 will be given for each with 5 being low risk and 1 being high risk. The score of both will then be added together, the lower the overall score the more serious the risk is to be considered. When this occurs actions should be identified in order to help reduce the risk occurring and reducing the impact if the risk did occur.
Project risks identified during test planning should be documented in the IEEE 829 test plan.
Product Risks
Examples of product risks are:
Error-prone software delivered
Poor requirements leading to badly defined and built software
Potential that a defect in the software/hardware could cause harm to an individual or company
Poor software quality characteristics (e.g. functionality, security, reliability, usability, performance) leading to poor user feedback.
The software does not meet the requirements and delivers functionality that was not requested
TEST ORGANISATION
Test organisation and independence
All humans make errors but's it's more difficult for the author of those errors to see them. In terms of software development, when a developer delivers code, they believe it to be complete and correct, however, when testers receive the code they believe it to contain defects and seek to expose those defects through testing.
Independent testing is important as it is generally harder for authors to see their own mistakes but is easier for other to spot them. There are several levels of independence.
Levels of independence from Low to High
The author/developer
Testers within the same development team
Independent test team within the organization
Independent test team provided by the operational business units
Specialist test such as usability testers, security testers, performance testers
Outsourced test teams or testers.
Independence also comes at a price. The greater the level of independence, the greater the level of bugs being raised through unfamiliarity.
In the real world, organizations and teams will vary in regards to roles and the level of testing independence which can be maintained. For software which is considered safety critical, there will be a need for multiple levels of testing by various teams and testers.
It's important that a tester is able to switch to an independent view of the software and not fall into the same mind set as developers "who own the product". This can be achieved by not assuming anything and ignore the view of "That was how it was developed to work".
Benefits
The tester sees other and different defect to the author
The tester is unbiased
The tester can see what has been built rather than what the developer thought had been built
The tester makes no assumptions regarding quality
Drawbacks
Isolation from the development team, which will mean the tester will be totally dependent on the test basis to understand what needs to be tested.
The tester may be seen as the bottleneck, as independent test execution is usually carried out at the last stage and is affected by any delays earlier in the process
Developers lose a sense of responsibility for quality as it may be assumed that they need not worry about errors because the testers will find them.
The fully independent view sets developers and testers on opposite sides of the fence. This can be a hindrance to communication and common understanding and effective working.
Tasks of a test leader and tester
Test tasks are usually carried out by people who make testing a career, however, test tasks may also be carried out by non-tester such as a project manager, developer and business and domain experts.
The two main testing roles (as covered in the book) are tester and test lead. These roles can be undertaken by anyone who possesses the required skills.
The tasks are undertaken by a test leader align very closely with those undertaken by a project manager and align closely with standard approaches to project manament. In this context, a test leader is anyone who leads a team of testers. Other names for a test leader are Test program manager, test manager, test team leader and test coordinator.
Typical test leader tasks include:
Coordinating the development of the test strategy and plan with project managers and others
Writing or reviewing test strategies produced for the project, and test policies produced for the organization
Contributing the testing perspective to other project activities such as development delivery schedules
Planning the development of the required tests
Managing the specification, preparation, implementation, and execution of tests, including the monitoring and control of all the specification execution.
Taking the required action, including adapting the planning, based on test results
TEST PLANNING AND ESTIMATION
Test Planning
Test planning is the most important activity undertaken by a test leader in any project. It ensures that there is initially a list of tasks and milestones in a baseline plan to track progress against, as well as defining the shape and size of the test effort.
Test planning will be used in green field projects as well as maintenance\support projects.
The main project used in test planning is often called a master test plan or a project test plan. The test plan is normally produced during the early stages of a project and will provide sufficient information to enable a test project to be established.
The details of the test-level activities are documented within test level plans, e.g the system test plan. These documents will contain the detailed activities and estimates for the relevant test levels.
A useful aid to help remember the 16 sections the IEEE 829 test plan is the acronym "SPACEDIRT", each letter mapping to one or several sections of the test plan
S - Scope (Including test items, features o be tested and features not to be tested)
P - People (Including responsibilities, staff and training approvals)
A - Approach
C - Criteria (Including item, pass/fail criteria and suspension and resumption requirements)
E - Environment needs
D - Deliverables (test)
I - Identifier and introduction (test plan)
R - Risks and contingencies
T - Testing tasks and schedule
Test Planning Activities
During test planning various activities have to be undertaken by those working on the plan:
Putting together the overall approach of testing, ensuring that test levels and entry and exit criteria are defined
Liaising with the project manager and making sure that the testing activities have been included within the software life-cycle activities such as:
Design - The development of the software design
Development - The building of the code
Implementation - The activities surrounding implementation into a live environment
Working with the project to decide what needs to be tested, what roles are involved and who will perform the test activities, planning when and how the test activities should be done, deciding how the test results will be evaluated and defining when to stop testing (exit criteria)
Finding and assigning resources to the different tasks that have been defined
Deciding what the documentation for the test project will be, eg which plans, how the test cases will be documented etc.
Defining the management information, including the metrics required and putting in place the processes to monitor and control test preparation and execution, defect resolution and risk issues.
Ensuring that the test documentation generates repeatable test assets eg test cases.
Exit Criteria
Exit criteria are used to determine when a given test activity has been completed or when it should stop. Exit criteria can be defined for all of the test activities, such as planning, specification, and execution as a whole or to a specific test.
Exit criteria should be included in the relevant test plans.
Some typical exit criteria might be:
All tests planned have been run
A certain level of requirements coverage has been achieved
No high-risk areas have been fully tested, with only minor residual risks left outstanding
Cost - when the budget been spent
The schedule has been achieved, eg the release date has been reached and the product has to go live This was the case with the Millennium testing (it had to be completed before midnight on 31 December 1999) and is often the case with government legislation.
Exit criteria should have been agreed as early as possible in the life cycle, however, they can be and often are subject to controlled change as the detail of the project becomes better understood and therefore the ability to meet the criteria is better understood by those responsible for delivery.
Page 6
Check of Understanding
CHAPTER ONE
Describe the interaction between errors, defects, and failures:
Errors lead to defects which the lead to failures
Software failures can cause losses. Give three consequences of software failures:
Loss of money
loss of reputation
Loss of life
What are the vertices of the "triangle of resources":
--------------------------------------------------------
Time
Features
Money Quality
---------------------------------------------------------
Describe static testing and dynamic testing:
Static testing is testing areas of the dev cycle such as testing the requirement specification for errors and ambiguities.
Dynamic testing is performing testing on the application with using test data.
Debugging is when a developer will try to find the cause of an error and apply a fix
What other elements apart from test execution are included in testing?
Test Planning
Documenting Test Results
Why are "Zero defects" an insufficient guide to software quality?
Finding no defects during test execution doesn't mean that there are no defects in the application, it only means that there were no defects found in that execution which could potentially represent an insufficient test plan.
Give three reasons why defect clustering may exist?
System Complexity
Volatile Code
Effects fo change upon change
Developer Experience
Developer Inexperience
Briefly, justify the idea of early testing
By finding defects earlier in the SDLC, costs are reduced as it is far more expensive to find these defects later in the process.
What are the stages in the fundamental test process?
Test Planning & Control
Test Analysis & Design
Test Implementation & Execution
Evaluating Exit Criteria Reporting
Test Closure
Briefly, compare regression testing and confirmation testing
Confirmation testing is testing around an area of work where a bug has been raised and fixed. Its purpose is to confirm that the error where the error was found is now working as expected.
Regression testing is testing around the other areas where a bug was raised to confirm that while fix hasn't caused issue in these other areas
When should the expected outcome of a test be defined?
The expected outcome should be defined in during the analysis and design and used to check that actual outcome matches the expected outcome
When testing software who has the highest level of independence?
Testers who are not a part of the same company doing the development such as a consultancy firm.
Contrast the advantages and disadvantages of developers testing their own code:
Advantages
Able to apply fixes straight away
No need for extensive error logs
No need to raise bugs, test cases etc
Disadvantages
Developers can be blind to mistakes in the code they have written
Developers will modify their behavior to fit project deadlines and objectives, this can lead to incomplete or inadequate testing
Suggest three ways confrontation can be avoided
Always treat errors and defects as software issues, not mistakes a person has made
Keep focus on delivering a quality product
Communicate effectively
CHAPTER TWO
What is meant by verification?
Verification is ensuring the application matches the requirements defined for it.
What is meant by Validation?
Validation is ensuring the application matches the end-users needs.
Name three work products typically shown in the V-model
Requirements Specification, Functional Specification, Technical Specification
Name three activities typically shown in the V-model
Requirements Testing, Functional Testing, Technical Testing
Identify a benefit of the V-Model
The main benefit is that we have solid sets of specifications to test against which makes testing easier.
Name a drawback fo the V-Model
One drawback is, each stage must be completed before the other is able to start, if the requirements are altered or changed part way through, this may not be caught until near the end when testing starts which could be costly.
Name three activities typically associated with the iterative model
Requirements gathering, Design, Coding.
Identify a significant benefit of the iterative model
Being able to test after each iteration offers the benefit of being able to capture issues and fix them in smaller time scales which reduces costs.
List three challenges of an iterative model.
No set and definitive requirements to test against.
Requirements can be changed or adjusted without being documented which could cause confusion.
More regression testing will be required to ensure all parts of the application work following each iteration.
List three types of iterative development.
Agile development, Rapid Application Development, Prototyping.
In the V-Model, which document would be used as the test basis for unit testing?
The technical specification.
Describe three typical integration testing strategies
Big Bang - Testing all components at once (not advisable)
Top-Down The units which will call other are tested first. If a unit to be called has not yet been developed stubs are created.
Bottom-Up The units which receive calls are tested first. If a calling unit has not yet been tested drivers are created.
Identify why stubs and drivers are usually used
As above.
In the V-Model, which document is used as the test basis for system testing,
The Functional Specification
Compare a functional requirement with a non-functional requirement
Functional Requirement is an operation the system must perform such as the ability to add items to a basket
Non-Functional Requirements are generic operations which can usually be applied to most applications such as a system responding to inputs within a reasonable time manner
List Three non-functional requirements
Performance
Maintainability
Usability
What is the purpose of acceptance testing?
The purpose of acceptance testing is to ensure the application meets the users expected requirements and expectations
In the V-model, what is the test basis for acceptance testing?
The Requirments Specification
Identify three types of acceptance testing.
Operational
User
Contract
CHAPTER THREE
Identify three benefits of reviews
Development productivity can be improved and timescales reduced
Improved communication
Testing costs reduced
What happens during the planning phase of a review?
Selecting personnel
Allocating Roles
Defining the entry and exit criteria
Selecting parts of documents to be reviewed
Who manages the review process?
The Manager
Compare the differences between a walk through and an investigation
Name three characteristics of a walk through
Identify at least five success factors for a review
----
What is static analysis looking for?
Looking for defects in the source code and software models
Name the five benefits of static analysis
Earlier detection of defects prior to test execution
Early warning about suspicious aspects of code such as complex functionality
Improved maintainability of code and design
What is the name of the type of defect which relates to password checks?
Security Vulnerability
What defines the process of test execution?
Briefly, compare a test case and a test condition
The test condition is defining what it is we could expect to see based on the test case.
The test case is what will be needed to test the defined
Which document identifies the sequence in which test is executed?
Describe the purpose of a test coverage measures?
CHAPTER FOUR
What is the relationship between a partition and a boundary?
We can use the higher and lower boundaries of an input to partition it's valid or invalid values and use these as or test conditions which we will test against.
Explain what is meant by "as close as possible to a boundary"?
If we have a boundary with a lower of 1 and a higher value of 100, to test the valid conditions which are "as close as possible to the boundary", we would select 3 valid values which would be 99,1001 and 2.
For the invalid value, we do the same using the closest invalid values of -1, 0, 101, 102.
What is the main use of ST for tester
To quickly analysis the various states and events and use it to write up test cases
Name three components of a state transition diagram
State, Event & Transition
How are negative tests identified from an ST
Using a null value to indicate that no transition will take place on that transition.
What is meant by the term "invalid transition"?
This is when the event it self-has no effect and is marked as Null (N)
What is the purpose of a use case?
Use cases allow us to determine the different types of users who will interact with a system and identify test cases based on the user actions.
What is the relationship between a use case and a test case?
A use case helps us identify specific actions to use for a test case.
What is meant by the term executable statement?
Executable refers to code which instructs to the computer to perform some type of action.
Describe the two forms of looping structure so far explained
Do While
This is a type of iterations that says while a condition is true keeping looping otherwise stop
Repeat
This is a type of iteration which behaves similarly to the do while but the condition is at the end meaning the statements inside the loop will always be executed at least once.
What is a selection structure?
A selection structure checks if a condition is true or false using and if, else statement.
How many different paths are there through a selection structure
At least one path but there could be much more
What is the difference between a flow chart and a control flow graph?
A flow chart is used to represent the structure of a program, which captures statements and decisions.
A control flow graph is used to represent only the decisions of a code segment.
Name the three fundamental program structures that can be found in programs?
Sequence
Selection
Iteration
Briefly explain what is meant by an edge, a node and a region in a control flow graph.
Node - Represents any point where the code can be modified
Edge - A line which connects any two nodes
Region - This is the closed region of a collection of nodes and edges
What is the purpose of decision testing?
To determine how many test cases will be required to ensure a thorough enough test has been carried out
How many test cases are needed to exercise a single decision
It will depend on how many statements are then executed from that decision.
How many test cases are needed to exercise a loop structure
It will depend on how many possible decisions there are in the loop structure.
CHAPTER FIVE
What are the two types of risks that have to be considered in testing?
Project risks and product risks.
Compare and contrast these two risk types
Project risks are risks which could have an impact on the delivery of the project it self
Product risks are risks which have an impact on the actual product being built
How early in the life cycle can risk impacting the testing approach?
From the very start
What does MISRA determine when the level of risk is understood
It determines the level of test coverage which should be undertaken
Why is independent testing more effective at finding errors than simply allowing the developer and author to test their own product.
Becuase authors tend not to see mistakes in their own work or something they have ownership in.
Name three benefits of independence.
The tester is unbiased
The tester will find other/different defects to the author
The tester will make no assumptions regarding quality
Which organization provides the lowest level of independence and which provides the highest.
Small organizations where the team take on several roles
Organization which is developing an application which is safety critical
Page 7
EXAMPLE EXAMINATION QUESTION WITH ANSWERS
The Fundamentals of Testing
E1. K1 Question - Which of the following is correct: Debugging is:
C. Identifying the cause of a defect, repairing the code and checking the fix is correct.
E2. K2 Question - Which of the following are aids to good communication and which hinder it
Answer is C
E3 K1 Question - Which option is part of the implementation and execution area of the fundamental test process
B Comparing actual and expected results
E4 K1 Question - The five parts of the fundamental test process have a broad chronological order. Which of the options give three different parts in the correct order
Answer is B
E5 K2 Question - Which pair of definitions is correct?
D. Regression testing is checking that there are no additional problems in previously tested software, confirmation test is demonstrating that the reported defect has been fixed.
E6 K1 Question - Which statement is most true
A. Different testing is needed depending upon the application.
E7 K1 Question - When is testing complete
D. When every data combination has been exercised successfully
E8. K1 Question - Which list of levels of tester independence is in the correct order. starting with the most independent first
Answer is C
E9. K2 Question - The following statements relate to activities that are part of the fundamental test process
Answer is B
Score 8/9 - One Incorrect answer for E7, answer should be B
Life Cycles
E1. K1 Question - Which of the following is usually the test basis for integration testing?
C. Technical specification
E2. K2 Question - A top-down development strategy affects which level of testing most?
B. Integration testing
E3. K2 Question - Which of the following is a non-functional requirement?
D. The system will allow up to 100 users to log in at the same time
E4. K1 Question - Which of the following are examples of iterative development models?
Answer is C
E5. K2 Question - Which of the following statements is true?
Answer is D
E6. K1 Question - Which of the following is not true of regression testing?
A. It can be carried out at each stage of the life cycle
Score 6/6 -
Static Testing
E1 K1 Question - Which of the following is most likely to be a benefit of using static techniques?
B. Productivity improvements in the development process
E2. K2 Question - Which of the following has the typical formal review activities in the correct sequence?
C. Planning, Kick-off, Individual Prep, Review Meeting
E3. K2 Question - Which of the following statements is true?
D.
E4. K1 Question - Which of the following are static techniques?
A. Walkthrough
E5. k1 Question - Which one of the following roles is typically used in a review?
B. Author
E6. K2 Question -Which of the following is most likely to be performed by developers
D. Static analysis of a software model.
Score 6/6
Test Design Techniques
E1. K1 Question - Which of the following scribes structure-based (White box) test case design techniques.
C. Test cases are derived systematically from the delivered code
E2. K1 Question - Which of the following is a structure-based (White box) technique?
B. State transition testing - Incorrect, should be C
E3. K3 Questions
A washing machine has three temperature bands for different kinds of fabrics: fragile fabrics are washed at temperatures between15 and 30 degree Celsius; normal fabrics are washed at temperatures between 31 and 60 degrees Celsus; heavily soiled and tough fabrics are washed at temps between 61 and 100 C. Which of the following contains only values that are in different equivalence partitions?
C. 25, 45, 75
E4. K3 Questions - Consider the following pseudo code:
B. 2
E5. K1 Question - What is the purpose of use case testing?
A. To identify defects in process flows related to tical use of the system.
E6. K2 Question - Which of the following are the most important factors to be taken into account when selecting test techniques
B.
Score 5/6 One Incorrect answer for E2, answer should be C
Page 8
Self-Assesment Questions
CHAPTER ONE
A bug defect is:
C. The result of an Error or Mistake
The effect of testing is to:
B. Give an Indication of the software quality
What is confirmation testing:
D. Running previously failed test against new software/data/documents to see if the problem is solved.
CHAPTER TWO
Which of the following is true about the V-Model?
D. It enables test planning to start as early as possible
Which of the following is true of iterative development?
B. It involves the users in the testing throughout
Which of the following is in the correct order (typically)?
A. Unit Testing, System Testing, Acceptance Testing, Maintenance Testing
Page 9
Why Software Fails
Error - > Defect - > Failure
Incorrect Software Can Harm
People
Companies
The Environment
Software Failure Can Lead To
Loss of money
Loss of time
Loss of business reputation
Injury
Death
Testing and Risk
Greater risk implies more and better testing
Testing and Quality
There will be a level of quality which will be considered acceptable for a given system. Testing doesn't increase the quality itself but instead finds and reports defects making it possible for them to be removed which contributes to the enhanced quality of the system.
Time
Features
Money Quality
Managing Testing Time
Not everything can be tested so we need to prioritize the first test the most important areas which will also address the most significant risks.
The next important aspect is to set criteria which will give an objective test of whether it is safe to stop testing, known as completion criteria.
Page 10
EXERCISES
Chapter 4 - Test Design Techniques
Exercise 4.2
Charge = £3.95 post for order upto £20
= £4.05 for orders above £20
= order above £40 free postage
Partitions
Valid
Less than £20 - Test using £9.99
£20 - £40 - Test using £25
Greater than £40 - Test using £100
Invalid
Zero
Negative Values (Real Numbers)
Decision Tables - Exercise 4.4
Answer: Eligible for cash payments
State Transition Tables - Exercise 4.5
Answer - b. FEABC
Page 11
UDEMY COURSE NOTES
Causes of Software Defects
Faulty Requirements Definition
Time pressures
Complex Code
Lots of interaction between systems
Coding Errors
Complexity of Infrastructure
Changing Technologies
Non-compliance with standards
Test Levels
Unit Testing
Testing the individual units of code, these would be the user stories in a sprint
Integration Testing
Testing how each of the units works together, individual units will be interacting with other units of code
System Testing
Testing the various scenarios and ways in which the system can be used
Acceptance Testing
Testing the software from a users perspective
Static Techniques and the Test Process
Testing software without executing code
Find causes of failures rather than the failures
Usually used before dynamic techniques
Complementary to dynamic techniques
Manual reviews and automated static analysis
Static Testing Types
Reviews
Static Tools
Specification-based Techniques
Equivalence Partitioning (k3)
Inputs are divided into partitions that are expected to exhibit similar behavior
One value from each partition should be selected
Equivalence partitions can be found for both valid data and invalid data
Boundary value analysis (k3)
Decision table testing (k3)
State transition testing (k3)
The state transition technique is concerned with systems that may exhibit a different response depending on current conditions or previous history
It allows the tester to view the software in terms of its states transitions between states, the inputs or events that trigger state changes (transitions) and the actions which may result from those transitions
State transition tables can be used to record all the relationships between the states and inputs which can highlight the possible transitions that are invalid
Creating state-transition tables often unearths combinations that have not been identified in the requirements. It is highly beneficial to discover these defects before coding begins.
Use Case Testing (k2)
Risk and Testing
Risk can be defined as the chance of an event, hazard, threat or situation occurring and resulting in future negative consequences or a potential problem
A risked-based approach to testing provides proactive activities to reduce the risk levels of product risk, as early as possible in the SDLC.
Risks are used to decide where to start testing and where to test more
Risk Management activities:
Risk Identification
Risk Analysis
Risk Response
Risk Analysis
The level of risk will be determined by the likelihood of an adverse event happening and the impact
LEVEL OF RISK = (Probability of the risk) X (Impact if it happens)
Risk Responses
Avoid
Doing whatever can be done to make the probability and impact of the rise zero
Mitigate
Lowering the probability and impact of the risk
Transfer
If possible move the risk from your being your responsibility to someone else
Accept
Having a plan in place which will handle the risk thus reducing the overall impact of that risk
Project Risks (k2)
Technical Issues
Problems in defining the right requirements
Test environment not ready
Old testing tools
Low quality of test basis
Late test basis
Organisational Factors
Skills, Training
Personal Issues
Political Issues
Improper Attitude
Supplier Issues
Failure of a third party
Contractual; issues
Product Risks (K2)
Failure Prone Software
Software/Hardware could cause harm to an individual or a company
Software that does not perform it's intended functions
Poor Software Characteristics
Poor data integrity and quality
Page 12
Test Fundamentals - UC
Software Testing Causes
Faulty Requirements Defenition
Time Pressure
Complex Code
Many System Interactions
Coding Errors
Complexity of Infrastructure
Changing Technologies
Non-Compliance with Standards
Testing & Quality
Testing ensures that key functional and non-functional requirements are met
Testing measures the quality of software in terms of the number of defects found, the tests run and the systems covered by the tests
Testing doesn't directly increase the quality of the software under test but instead, give us an accurate indication of the level of quality of that software, if few or no defects are found this can increase confidence in the quality of the software.
Understanding the root cause of issues found in projects can help prevent those same issues re-occurring in other projects
Testing is just one component of the overall quality assurance activity
Testing Objectives
Finding Defects
Gaining confidence about the level of quality
Providing information decision making
Preventing Defects
Testing Principles
There are seven testing principles:
Testing shows the presence of defects
Cannot prove that there are no defects
Tests should be designed to find as many defects as possible
Exhaustive testing is impossible
Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases
Instead of exhaustive testing, risk analysis and priorities should be used to focus testing efforts
Early Testing
The earlier defects are spotted the less the impact of cost is
Testing activities shall be started as early as possible in the software or system development life cycle and should be focused on defined objectives.
Defect clustering
A small number of modules usually contain most of the defects discovered during pre-release testing or are responsible for most of the operational failures
Pareto Principle says 80 / 20 (read further about this)
Pesticide Paradox
If the same tests are repeated over and over again, eventually the same set of test cases will no longer find any new defects
Test cases need to be regularly reviewed and revised
Testing is context dependent
Testing is done differently in different contexts
Risk can be a large factor in determining the type of testing that is needed
The absence of errors fallacy
Finding and fixing defects does not help if the system built is unusable and does not fulfill the users' needs and expectations
How much testing is enough? (This is the type of question which could be asked)
The answer to this question is that it really depends on the risk of:
Missing important faults
Incurring failure costs
Releasing untested or under-tested software
Losing credibility and market share
Missing a market window
Over-testing, ineffective testing
"Testing should provide sufficient information to stakeholders to make informed decisions"
Risk can then be used to determine:
What to test first
What to test Most
How thoroughly to test each item
What not to test (this time)
Allocate the time available for testing by prioritizing testing
Fundamental Test Process
The fundamental test process consists of the following activities
Planning & Control
Test planning is the act of defining the objectives of testing and the specification of test activities in order to meet the objectives and mission
Test control is the ongoing activity of comparing actual progress against the plan and reporting the status, including deviations from the plan.
Analysis & Design
Test analysis and design is the activity during which general testing objectives are transformed into tangible test conditions and test cases
Implementation and Execution
Test implementation and execution is the activity where test procedures or scripts are specified by combining the test cases in a particular order and including any other information needed for test execution, the environment is set up and the tests are run
Evaluating Exit Criteria and Reporting
Evaluating exit criteria is the activity where test execution is assessed against the defined objectives
Test Closure
Test closure activities occur at project milestones such as when a software system is released, a test project is completed or cancelled or when a maintenance release has been completed
Levels of independent testing
A certain degree of independence often makes the tester more effective at finding defects and failures
The further away a tester is from the owner for the software under test, the more effective that testing will be, it's difficult for a developer to test their own code and there more likely to be blind to the mistakes in their own work.
IEEE-CS/AM Software Engineering Code of Ethics and Professional Practice
Public
Software engineers (SE's) shall act consistently with the public interest
Client & Employer
SE's shall act in a manner that is in the best interest of their client and employer consistent with the public interest
Product
SE's shall ensure that their products and related modifications meet the highest professional standards possible
Judgment
SE's shall maintain integrity and independence in their professional judgment
Management
SE managers and leaders shall subscribe to an promote an ethical approach to the management of software development and maintenance
Profession
SE's shall advance the integrity and reputation of the profession consistent with public interest in mind
Colleagues
SE's shall be fair to and supportive of their colleagues
Self
SE's shall participate in lifelong learning regarding the practice of their profession and shall promote an ethical approach to the practice of the profession
Page 13
Testing Throughout The Software Life Cycle
Software Development Models:
Guidelines
At every stage, a check should be made in order that the work-product for that stage meets its objectives
Work-product evaluation should take place when the product has been declared complete
If the work-product passes its evaluation (test), we can progress to the next stage in confidence
Fixing any problems at the point of creation is cheaper than fixing them at a later stage
Verification and Validation (VV & T)
Verification
Determine whether the work product satisfies the conditions imposed at the start of that phase
Validation
Determination of the correctness of the work product with respect to the user needs and requirements
Testing
The process of exercising software to verify that it satisfies specified requirements and to detect faults
The V-Model
Early test planning and design should be carried out throughout the software development lifecycle
See image:
Iterative-incremental Development Models
There is a difference between incremental and iterative life cycles (do some reading).
Incremental will tend to have a defined requirement at the start where specific sections of the overall project will be completed and added on each iteration.
Iterative development will typically have no solid requirement, to begin with, but the business/user will provide feedback at the end of each iteration which will help mould the project requirements.
It is also possible that many projects will utilize a mix of both Incremental and iterative development
Prototyping, Rapid Application Development, Rational Unified Process (RUP) and Agile are all form of iterative development life cycles
In any life cycle model, there are several characteristics of good testing
For every development activity, there is a corresponding testing activity
Each test level has test objectives specific to that level
The analysis and design of tests for any test level should begin during the corresponding development activity
Testers should be involved in reviewing documents as soon as drafts are available in SDLC
Change Control System - Further Reading here https://www.guru99.com/change-control-business-analyst.html
Testers involved in Iterative models need to have some sort of change control system put in place. This allows testers to keep an eye on software changes as they happen
Test Levels
Acceptance Testing
Acceptance testing is often the responsibility of the customers or users of a system
The goal of acceptance testing is to establish confidence in the system
Finding defects is not the main focus
Acceptance testing may occur at various times in the life cycle for example:
A COTS software product (Commercial Of The Shelve)
A component
Systems Testing
System testing is concerned with the behaviour of a whole system/product
The test environment should correspond to the final target or production environment as much as possible in order to minimize the risk of environment-specific failures not being found in testing.
Typically carried out by an independent test group
Integration Testing
More than one (tested) component
Testing interface and communication between components
Done by designers, analysts, or specialist integration testers.
Unit Testing
Lowest level of testing
Also known as component, module or program testing
Usually performed by the developer/programmers and often are not logged or recorded but instead fixed instantaneously
Unit testing is intended to ensure that the code written for the unit meets it's specification, prior to its integration with other units
Usually tested in isolation using stubs and drivers.
Stubs and Drivers
Typically when a developer is working on a new user story there will be areas of functionality which will need to be tested but the other segments of code which interacts with the written unit have not yet been written.
In order to get around this developers will ofter create whats know as stubs or drivers, which small bits of functional code which mock the missing parts.
A stub is a segment of code which will be called by the unit,
A driver is a segment of code which will make calls to the unit
Integration Testing
Test basis includes
Software and System Design
System architecture
Workflows
Use-cases
Test objects include
Sub-system database implementation
Infrastructure
Interfaces
Integration Types
Big bang integration
Incremental integration
Top-down integration
Bottom-up integration
Functional integration
It's possible for there to be different levels of integration testing
Unit Integration Testing
Tests the interactions between software units and is done after unit testing
Usually done by developers
Systems Integration Testing
Tests the interactions between different systems and may be done after system testing
Usually done by testers
Testers should understand the architecture and influence the integration planning
System Testing
Test basis includes
System and software requirement specification
Use Cases
Functional specification
Test objects include
System, user and operation manuals
System configuration
Configuration data
Acceptance Testing
Test basis includes
User requirements
System requirements
Use cases
Business processes
Risk analysis reports
Test objects include
Business processes on fully integrated system
Operational and maintenance processes
User procedures
Forms
Reports
Configuration data
Acceptance Testing Types
User Acceptance Testing
Typically verifies the fitness for use of the system by business users
Customer (user) should perform or be closely involved
Customer can perform any test they wish, usually based on their business processes
Can happen at any time during the project
Before the final user sign-off
Operational Acceptance Testing
The acceptance the system by the system administrators
Testing of backup/restore
Disaster recovery
User management
Contract and Regulation Acceptance
Testing is against the contract and any documented agreed changes
Testing is performed against any regulations that must be adhered to, such as government, legal or safety regulations.
Alpha & Beta Testing
Alpha testing is performed at the developing organization's site but not by the developing team
Beta testing, or field-testing, is performed by potential customers at their own locations
TEST TYPES
Functional Testing
The functions are "what" the system does
May be described in word products such as a requirements specification, use cases, or a functional specification.
Types of functional testing include
Security Testing
Investigate the functions relating to detection of threats, such as viruses from malicious outsiders
Interoperability Testing
Evaluates the capability of the software product to interact with one or more specified components or systems
Non-Functional Testing
Is the testing of "How" the system works
Non-functional testing includes but is not limited to
performance testing
How many users can connect to the system and how will that affect the performance of the software
load testing
How will the system perform under a heavy number of concurrent users, performing transactions over a certain period of time? How can the system handle heavy load volumes?
stress testing
How will the system perform extreme and intense loads, how does it recover when going back to normal, do KPI's, throughput and response times remain the same as before. Load testing can also expose memory leaks, slowness, security issues and data corruption.
usability testing
Is the system easy to use
maintainability testing
Is the system easy to maintain if we need to fix a defect
reliability testing
Is the system reliable or does it crash eventually
portability testing
Is the system easy to port from one platform to another
Structural Testing (AKA Wide box Testing)
Coverage is the extent that a structure has been exercised by testing, expressed as a percentage of the items being covered
If coverage is not 100%, then more tests may be designed to test those items that were missed to increase coverage
Testing Related to Changes
After a defect is detected and fixed, the software should be retested to confirm that the original defect has been successfully removed. This is called CONFIRMATION testing or Re-Testing.
REGRESSION testing is the repeated testing of an already tested program, after modification, to discover any defects introduced or uncovered as a result of the changes
Maintenance Testing
Testing which takes place on a system which is in operation in the live environment is called maintenance testing
Triggers of Maintenance Testing
Modification
Planned enhancement changes
Corrective and emergency changes
Changes in environment
Migration
From one platform to another
Conversion testing
Testing of data migration
Archiving the data
Includes Extensive Regression Testing
Things to consider
What could this change have an impact on
How important a fault in the impacted area
Test what has been affected, but how much
Most important affected areas
Areas most likely to be affected
Whole system
Page 14
Static Techniques and the Test Process
KiStatic Techniques allow us to
Test software without executing the code
Find causes of failures rather than the failures
Usually used before dynamic techniques
Complementary to dynamic techniques
Manual reviews and automated static analysis
STATIC TESTING TYPES
Reviews
A review is a systematic examination of a document by one or more people with the aim of finding and removing errors
Reviews can be used to test anything that is written or typed; this can include documents such as requirement specifications, system designs, code, test plans and test cases
Static Tools
BENEFITS OF REVIEWS
Early feedback on quality issues can be established
Development productivity improvement
Improved team communication and knowledge transfer
Lifetime cost reductions
Reduced fault levels
Reduced development timescales
Reduced testing time and cost
TYPICAL DEFECTS FOUND IN REVIEWS
Requirement defects
Design Defects
Insufficient Maintainability
Incorrect interface specifications
Deviations from standards
REVIEW PROCESS
Activites of Formal Review
Planning
Plan the review activities
Selecting the personnel
Allocating roles
Defining entry & exit criteria
Entry - Does it contain a minimum set of specifications and categories
Exit - What should be met at a minimum, before exit ie. identified bugs in requirement fixed
Kick-off
Distributing documents
Explaining the objectives and process
Can be run through a meeting or just sending an email
Checking entry criteria
To ensure that the reviewers time is not wasted on a document that is not ready for review
Individual preparation
Reading documents before meetings etc should be time-boxed 2-4 hours
Noting incidents
Should be time-boxed
Logging potential defects suggestions and questions
Should watch for the checking rate which is the number of pages checked per hour
Review meeting
Discussing defects
More defects will be found by discussion
Just recommendations on how to fix a defect
Every defect and it's severity should be logged. The participant who identifies the defect proposes the severity
The approach of the meeting depends upon:
Time available
Review objectives
Type of review
Examine
Analyze the recording of the review output
Rework
Checking the logged defected and answer questions
Fixing defects
Update the status of the defects
Follow-up
Assure defects fixed
Collect metrics, such as time spent on review and number of defects found
Checking exit criteria
To officially close the review process for that document
Formal Review Roles and Responsibilities
Author
Leader / Moderator
Reviewers / Inspecters
Managers
Scribe / Recorder
Factors Effecting Review Formality
The maturity of the development process
Legal or regulatory requirements
The need for an audit trail
Review Types
Informal (No Process, Quick Benefit)
Informal process
May take the form of pair programming
Results usually not documented
Varies in usefulness depending on the reviewers
An inexpensive way to find defects and get some benefit
Walkthrough (Led by the author, Learning gaining understanding & optional activities)
Meeting led by author
Optional process
May vary in practice from quite informal to very formal
Main purposes: Learning, gaining understanding, gathering information & finding defects
Technical Review (Led by trained moderator, Discussing, Taking decisions and Evaluating alternatives, Most activities are executed, )
Ideally led by a trained moderator
Mostly follow process
May vary in practice from quite informal to very informal
Main purposes: discussing, making decisions & evaluating alternatives
Inspection (Led by trained moderator, Finding Defects, All activities are executed )
Led by trained moderator
Formal process
Main purpose: finding defects
Success Factors
Each review has clear predefined objectives
The right people for the review objectives are involved
The review is conducted in an atmosphere of trust
Will not be used for the evaluation of the participants
Defects found are welcomed and expressed objectively
People issues and psychological aspects are dealt with
Checklists or roles are used if appropriate
Traning is given in review techniques
Management supports a good review process
Testers are welcome as reviewers
Emphasis on learning and process improvement
Static Analysis Tools
Find defects rather than failures
Analyze program code and software models
Typically used by developers
Ideally performed before the types of formal review
Types of defects found would be hard to find during test execution
Can also be used to calculate code metrics
Value of Static Analysis Tools
Early detection of defects prior to test execution
Early warning about suspicious code or design
Identification of hard to find defects
Detecting inconsistencies in software models
Improved maintainability of code and design
Prevention of defects
Typical Defects Discovered By Static Analysis Tools
Page 15
Fundamental Test Process
Type in The Fundamental test process consists of the following activities
Planning & Control
Analysis & Design
Implementation and Execution
Evaluating Exit Criteria and Reporting
Test Closure
Analysis & Design
The analysis is to: Decide on a test condition, a test condition being a work item or event of a component or system that could be verified by one or more test cases, e.g. A function, transaction, feature, quality attribute or structural element.
Each test condition will be located in the Test Design specification document
The test condition will need to be prioritized
The design is to: Design a test case that will verify the test condition(s) or test objective(s)
Test Cases
A set of input values, preconditions and expected result and postconditions developed for a particular objective or test condition.
It's possible for a test case to cover more than one test condition
All test cases should be located in the Test Design specification document
Each test case should be prioritized
Implementation and Execution
Implementation involves writing a test procedure to execute the test and writing out a test execution schedule.
The test procedure outlines each of the steps to be taken in order to carry out the execution of test cases or cases against the test conditions
Test procedures should be located in the test procedure specification document, and need to be prioritized
Each procedure should be prioritized so that testing is carried out in an effective manner saving both time and cost
Execution Involves running through each of the test procedures, logging the results and reporting any discrepancies to the developers, project manager, and other interested parties.
-- EXAMPLE -------------------------------------------
Test Condition
REQ 1.A - The age of the employee shall be in the range from 20 to 50, whole numbers only
Test Case
Test Range 20 - 50
10
40
60
Test Whole Numbers
40
1.5
Test Numbers
40
xyz
$%^
Test Procedure One
Launch the "ExpertWave" application
Select the "New Employee" menu item from the "Employee" menu, the "New Employee" dialog should be displayed
Check that the cursor blinking inside the "age" field
Type in '40' and click the 'OK' button
Check that the message "Thank you" is displayed
Test Procedure Two
Launch the "ExpertWave application"
Type Ctrl-N, the "New Employee" dialog should be displayed
Check that the cursor blinking inside the age field
Type in '30.4' and press the Enter key
Check that the Alert "Age should be whole numbers only" is displayed
-- EXAMPLE END --------------------------------------------------
CATEGORIES OF TEST DESIGN TECHNIQUES
What is a testing technique?
The purpose of a test design technique is to identify test conditions, test cases, and test data
Effective Testing
Finding more faults
Focus attention on specific types of fault
Know you're testing the right thing
Efficient Testing
Find faults with less effort
Avoid duplication
Systematic techniques are measurable
Categories (here but are many more)
Specification-based test design techniques (Black Box)
Test cases derived systematically from Models, either formal informal
Structure-based test design techniques (White Box)
Test cases derived from information about how the software is constructed (e.g, Code or Database)
Experience-based test design techniques (User Testing)
The knowledge and experience of people are used to derive the test cases
Specification-based testing (Black Box)
Equivalence partitioning
Inputs are divided into partitions that are expected to exhibit similar behavior
One value from each partition should be selected
Equivalence partitions can be found for both valid data and invalid data
Boundary value analysis
boundaries are an area where testing is likely to yield defects
Tests can be designed to cover both valid and invalid boundary values. When designing test cases, a test for each boundary value is chosen
This technique is often considered as an extension of equivalence partitioning
Boundary value analysis can be applied at all levels
Decision table testing
is used by creating a table which contains action and condition columns which contain each potential scenario.
State transition testing
The state transition technique is concerned with systems that may exhibit a different response de[ending on current conditions or previous history
It allows the tester to view the software in terms of its states. transitions between stats, the inputs or events that trigger state changes (transitions) and the actions which may result from those transitions
A state table shows the relationship between the states and inputs and can highlight possible transitions that are invalid
Use case testing
A use case describes interactions between actors (users or external systems) and the system, which produces a result of value to a system user or the customer
Test cases derived from use cases are most useful in uncovering defects in the process flows during real-world use of the system
A use case usually has a most likely scenario and alternative path, precondition, postcondition and final state of the system after the use case has been completed
Use cases are very useful for deigning acceptance tests with customer/user participation
Are an excellent basis for system level testing
They also help uncover integration defects caused by the interaction and interference of different components
Designing test cases from use cases may be combined with other specification bases test techniques
Structure-based Testing (White Box)
Is based on the identified structure of the software or the system
Structure-based techniques serve two purposes
Test coverage measurement
Structural test case design
They are often used to assess the amount of testing performed by tests derived from specification-based techniques, i.e to assess coverage.
They are then used design additional test with the aim of increasing the test coverage.
Statement Testing and Coverage
Statement coverage is determined by the number of executable statements covered by (designed or executed) test cases divided by the number of all executable statements in the code under test.
If we test every executable statement we call this full or 100 percent statement coverage.
Statement coverage is the weakest white-box technique
Decision Testing and Coverage
Decision coverage, related to branch testing, is the assessment of the percentage of decision outcomes (e.g, the true and false options of an IF statement) that have been exercised by a test case suite
Decision coverage is determined by the number of all decision outcomes covered by test cases divided by the number of all possible decision outcomes in the code under test.
Decision coverage is stronger than statement coverage; 1005 decision coverage guarantees 100% statement coverage but not vice versa
Other sTructure-based
There are stronger levels of structural coverage beyond decision coverage, for example, condition coverage and multiple condition coverage
Coverage can also be applied at other test levels
Coverage can also be applied to black box testing techniques
Tool support is useful for the structural testing of code
Page 16
Test Strategy and Test Approach
TYPICAL TEST APPROACHES
The test approach is the implementation of the test strategy for a specific project.
A test approach includes all of the decisions made on how testing should be implemented, based on the (test) project goals and objectives, as well as the risk assessment.
It forms the starting point for test planning, selecting the test design techniques and test types to be employed. It should also define the software and test entry and exit criteria.
Analytical
Risk-based strategy
Requirements-based strategy
Model-Based
Stochastic testing
Methodical
Failure-based (including error guessing and fault attacks)
Checklist based
Quality-characteristic based
Process / standard-compliant
IEEE 829 standard
Agile methodologies (e.g. Extreme programming)
Dynamic and heuristic
Error guessing
Exploratory testing
Consultative
Ask the users or developers
Regression-averse
Test automation
Information
- Description
- Testing Note on 1 - The Fundamentals of Testing, created by Davie Crellin on 16/06/2017.
- Created by
- Davie Crellin
- Views
- 4
- Rating