Introduction to Software Testing

Page 1

Learning Objectives

Describe the key components of a software test and their purpose

Create automatable unit tests for java code in jUnit

Explain why software testing is particularly challenging within engineering disciplines

Explain the strengths and weaknesses of software testing for verification and validation

Compare and contrast verification and validation in the software development lifecycle

Explain the goal of testing

Page 2

Overview

Week 1: You will learn the basics of testing, the "what" of tests. What tests are and how to build them, with some hands-on tutorials on building your first set of executable tests.

Week 2: You will learn about the "why" of testing. What types of testing processes and techniques exist? How do we know what to use and when? And, how do we know our tests are any good?

Week 3: You will learn the process of testing. How do I organize the testing and document the results? How do I document and track defects that we (or our users) find in the software we build?

Week 4: Lastly, you will tackle the idea of Test Doubles, concepts that allow you to test your code without waiting. Instead of waiting for other code that your code relies upon to be built, we can "pretend" that the code is built, using some interesting software to mimic the behavior we rely on. This allows us to test our own code in isolation (proving our own code is right) without waiting for others (or future you) to build the other parts of the system. This will be a very hands-on module, putting into practice everything you've learned up to this point.

Page 3

Introduction

Why do we Test?

Testing is done to ensure that software functions in the way that it is intended and to ensure that bad things do not happen with the software, such as leaking customer information or deleting user information.

Cost of software failures

sometimes it's financial; other times, it's human lives.

Verification and Validation

Goal: To assure that the software meets user's needs.

Software Testing = Software Verification + Software Validation:

Verification

are we building the product right? Right means is the software built according to the specifications.

software should conform to the specifications

Validation

are building the right product?

the software should do what the user really requires

Techniques for Getting Software "Right"

need to understand and validate software requirements

V&V Techniques:

Inspections

Design discussions

Static analysis

Testing

Runtime Verification

Why focus on Software Testing?

the only software defect detection technique that can check the whole system

the current best way to assess some system behaviors accurately.

necessary for customer to accept the system.

a reasonably good way of documenting expected system behavior.

Testing is always incomplete, but the goal is to make it effective despite incompleteness.

Analysis Square

y - axis = complexity of property

x - axis = accuracy

Typical Testing (optimistic)

Simplistic Program Analysis (pessismistic)

Linting tools (lint)

false warnings

precise analysis of simple property

perfect verification of any property (infinite effort)

java type checking, haskell type checking, sound static analysis, unsound static analysis

Model Checking Temporal Properties

Tool-assisted automated testing

Coverage Testing

Theorem Proving

Page 4

Why is software testing challenging?

The Problem with Software Testing

inherently hard because it samples a set of possible behaviors.

discontinuous

there is no sound basis for extrapolating from tested to untested cases

all possible states of the system.

a system can have trillions of triillions of possible states.

What is required to find a bug?

knowledge of the system

knowledge of where programmers mistakes are likely to make:

Boundary Conditions/ boundary testing ****

Complex Boolean Expressions

Think like a helpful adversary - how can I break this code?

To examine well, we must:

examine and test from the requirements (black-box testing)

examine the code and test from the boundary conditions (white-box testing)

Testing from 10,000 feet

testing in terms of scale

Unit Test: start by testing small because we can test small things more rigorously.

Integration Test

Systems Test

Scale, process, purpose

Summary:

This section discussed why testing is so hard, which is mainly because a system has trillions upon trillions of possible states. Because of this, a tester has to be systematic and methodical when approaching tests. We should approach testing in terms of scale in the sense that we should test small batches of code first (unit tests), and then graduate to integration and systems tests. While thinking about these tests it is always helpful to think like a helpful adversary and know the system as well. Lastly, the most important thing is for a programmer to watch out for errors of complex boolean expressions and boundary conditions.

Page 5

What is a Test?

One sentence answer (OSA):

A test is a process that consists inputing data into a computing agent and observing the output to verify whether a unit of code is behaving as expected.

Software Under Test (SUT): whatever part or subset of

the program we have completed where we can exercise some behavior.

SUT is usually a module or unit of the code. Hence, a unit test.

A unit is some small block of code for which there is an expectation of how it should behave when it is ran. A unit can include method, routine, subroutine, or procedure.

Test:

Input

Output:

Oracle: something that does the comparison between the input data and the output data. Traditionally, it was a human. Now, it has been automated.

Anatomy of a Test:

A test case consists of test data (input) and expected output.

Setup

Test Cases:

Test Data (Input)

Expected Output

Invocation

SUT:

Actual Output

Assessment:

Oracle

Teardown (cleanup)

Thoughts: testing is carried out in many different forms but what is important to remember is that whatever block of code is being tested is known as the software under testing (SUT).

For example, if it is a unit of code under testing, then it is a unit test.

Furthermore, a test has a basic structure of 4 elements:

Setup: how you put the software under test into the state or conditions under which the test input would make sense.

Invocation: the execution of a single test case

Assessment: the act of observing the behavior of that software under test, potentially documenting the actions that were taken, and comparing the results with what was expected to happen.

Teardown: how you put the software under test into the state or conditions after which the expected output is observed

and, a test case consists of two elements: input ( test data) and output.

Page 6

On Coverage Testing

On coverage testing

On the "Analysis Square," I stated that coverage testing was likely more rigorous than the testing that developers commonly perform. Why do you think that coverage testing can lead to more rigorous test suites than just writing tests for common scenarios?

Testing coverage is an important metric that helps in determining the quality and effectiveness--both of which are important for the a successful application software.

On Test-Driven Development

Do you think test-driven development is a good idea? What might be a benefit of this approach? What might be a drawback?

To be honest, at this moment I am not sure. But, I do think that there is some good in a test-driven development because it gives you an idea of the conditions that your software needs to satisfy, which can give you an idea of what to make.

However, I can see the bad in the sense that just because your software pass tests does not mean that it will work because we can't know all possible states. Also, it is like the whole scenario in which a scientist sets out with a theory and creates experiements to prove his theory and thus he makes the day fit the theory and not the theory to fit the data. Likewise, programmers make the program fit the test instead of the test fit the program.

Page 7

Automation: Using a test framework

Page 8

Automation: Writing JUnit Tests

When writing code and then writing tests, you will write tests that will test the code and not the solution. It is the proverbial can't see the forest for the tree.

Test Driven Development

Behavior Driven Development

Pina Coladas Build Trust:

Plan

Create a goal:

Test-Driven Development

Behavior-Driven Development

Domain-Driven Development

Code

Build

Test