Plainware System, ACLware Ambulance

Michael Devon Jones

Plainware System, ACLware Ambulance

Software Project Test Plan

Test: ADS – Ambulance Dispatch System

Purpose

The purpose of this document is to outline the test strategy and overall test approach for the ADS project. This includes test methodologies, traceability, resources required, and estimated schedule.

Audience

The audience of this document is the project team and the project management team. This document is also written for the extended test team. The test lead, testers, and any outsourced testers should be able to utilize this document to understand the scope of work that must be accomplished by the test team. The document is intended to accomplish its purpose only for the intended audiences.

Revision History

Revision

Date

Updated By

Update Comments

0.1

Software Project Test Plan
Project: ADS – Ambulance Dispatch System
Purpose
The purpose of this document is to outline the test strategy and overall test approach for the ADS project. This includes test methodologies, traceability, resources required, and estimated schedule.

Audience
The audience of this document is the project team and the project management team. This document is also written for the extended test team. The test lead, testers, and any outsourced testers should be able to utilize this document to understand the scope of work that must be accomplished by the test team. The document is intended to accomplish its purpose only for the intended audiences.
Revision History
Revision
Date
Updated By
Update Comments
0.1
2007.07.16
Michael Jones
Initial document creation
0.2
2007.07.20
Yo
First Draft
0.3
2007.07.21
Susan Steven Segal
Ready for Review
0.4
2007.07.22
Ashley Miller
Final Revisions
1. Introduction
This section describes the objectives and extent of the tests. The goal is to provide a framework that can be used by managers and testers to plan and execute the necessary tests in a timely and cost-effective manner.

1.1 Test objectives
The objective of the test suite is to provide adequate coverage metrics, requirements validation, and system quality data such that sufficient data is provided for those making the decision to release.

1.2 Extent of tests
The tests referenced herein are written to validate use cases, requirements (both functional and non-functional), system architecture, and object design. The structured tests for object design will be run first as the components of the system are developed. The structured tests to validate the system architecture will be run next as the system is integrated in bottom-up fashion during integration test.
2. Relationship to other documents
This section explains the relationship of the test plan to the other documents produced during the development effort such as the RAD, SDD, and ODD (Object Design Document). It explains how all the tests are related to the functional and nonfunctional requirements, as well as to the system design stated in the respective documents. If necessary, this section introduces a naming scheme to establish the correspondence between requirements and tests.

2.1 Relationships to documents

Black box tests relating to use cases are developed from the use case diagram(s) in the RAD (requirements analysis document).

Black box tests derived from functional requirements are developed from the requirements lists in the RAD.

Performance tests derived from nonfunctional requirements are developed from the nonfunctional requirements in the RAD.

Structured (unit/white box) tests are generated from the OOD (Object Design Document). The specific tests are developed from the OOD component diagram of each of the components.

Integration tests are developed from the SDD (System/Architecture Design Document). The integration tests generally come from the overall package diagram describing the architecture of the system. The architecture is also used to help in determining the integration test approach. The test environment (hardware/software) is also derived from the SDD.

A visualization of the relationships to the other documents can be seen in the diagram below.
2.2 Test naming scheme

The names of test cases will indicate from where they have been derived using a system of prefixes. The following prefixes are use to denote the tests were derived from the following places.

uc_ tests derived from use cases
nfrs_ tests derived from nonfunctional requirements
arch_ integration tests derived from the system architecture spec
odd_ structured tests derived from the subsystem decomposition and component diagrams
e2e_ end-to-end systemic tests that exercise the entire user scenarios

If a test was derived from a particular requirement number or component design number, the test names will contain the requirement number or name after prefix followed by an underscore.

After the prefix and the number or name identifier, the test name shall contain a brief, but descriptive name.
3. System overview
This section, focusing on the structural aspects of testing, provides an overview of the system in terms of the components that are tested during the unit test. The granularity of components and their dependencies are defined in this section.

3.1 Software Architecture Overview

* Architecture Overview an excerpt from Deliverable 3: Architecture Specification
3.2 Subsystem Decomposition (Components)

* Subsystem Decomposition an excerpt from Deliverable 3: Architecture Specification

4. Features to be tested/not to be tested
This section, focusing on the functional aspects of testing, identifies all features and combinations of features to be tested. It also describes all those features that are not to be tested and the reasons for not testing them.

4.1 Features to be tested

Components developed in house

The components developed by this organization will be tested unless otherwise noted below in section 4.2.

Components developed by outsource vendors

Components outsourced to be developed specifically for this project where this test team has primary responsibility to test and validate those components will be tested.

Components outsourced to be developed specifically for this project where the outsourced vendor is responsible for development as well as testing will not be tested as components by the component testers. Rather, the test results from the vendor will be reviewed by the test leads, and if they pass review, the component will be tested in-house starting with integration test.
4.2 Items that will not be tested

3rd party and Off-The-Shelf components

It is assumed that 3rd party components were evaluated and the pros and cons properly weighed before choosing that component with our software. The interfaces to those components will be tested, but not the functionality or performance those components.

This includes any 3rd party websites and GPS devices or software.

Infrastructure components

The actual database software utilized is assumed to work as designed and will not be directly tested for functionality. Performance tests will be done during system test with respect to GUI response time that will involve the database. However, no testing will be done directly against the database.

The internet/WIFI backbone will be utilized during testing, however, no tests will be written executed to directly test the communications backbone.
5. Pass/Fail criteria
This section specifies generic pass/fail criteria for the tests covered in this plan. They are supplemented by pass/fail criteria in the test design specification. Note that “fail” in the IEEE standard terminology means “successful test” in our terminology.

5.1 Component Pass/Fail criteria

Tests executed on components only pass when they satisfy the signatures, constraints, and interfaces dictated by the Object Design Specification for that component. This includes positive tests, negative and stress tests, and boundary tests.

If a test exhibits a product failure to meet the objectives of the object design specification, it will fail and a defect/issue will be reported in the defect tracking system for review by the triage team.
5.2 Integration Pass/Fail criteria

Tests executed on integrated components only pass when they satisfy the signatures, constraints, and interfaces dictated by both the object design specification and the system architecture specification. This includes positive tests, negative and stress tests, boundary conditions, and tests that explicitly manipulate the interface environment (such as the physical connection to the database server).

If a test exhibits a product failure to meet the objectives of both the object design specification and the system architecture specification, it will fail and a defect/issue will be reported in the defect tracking system for review by the triage team.
5.3 System Pass/Fail criteria

Tests executed against the system use the functional requirements, non-functional requirements, and use cases as the oracle to determine pass or fail.

If a test exhibits a product failure to meet the objectives of any of the functional requirements, non-functional requirements, or the use cases, it will fail and a defect/issue will be reported in the defect tracking system for review by the triage team.
6. Approach
This section describes the general approach to the testing process. It discusses the reasons for the selected integration testing strategy. Different strategies are often needed to test different parts of the system.

6.1 General Test Strategy

Unit testing and component testing will be performed on the components as they are developed. Test will be executed using test code in the form of either custom test tools or as an automated suite of tests run against the components in their individual sandboxes.

Integrations tests will be performed by both the component testers as well as the system testers. The BAT and the unit test suite will be used as a regression during the integration of components. However, as the integration begins to include GUI level functionality, the tests being run will utilize significantly more manual testing and less automated testing.

System test will require a new set of tools that can measure NFRS compliance, such as LoadRunner (load testing) or FindBugs (java source code analysis for security issues). Manual tests will start by validating functionality based on the requirements. Later stages of system test will include manual end-to-end tests to validate use cases.

6.2 Integration Test Strategy

Because the components will be developed from the bottom-up and top-down, the test strategy will also align to the order of development of components. This will utilize a mostly bottom-up integration test approach, but will also involve the sandwich integration test approach.

Please review the system architecture overview and the subsystem architecture in section 3.

6.3 Test Case alignment with test phases

7. Suspension and resumption
This section specifies the criteria for suspending the testing on the test items associated with the plan. It also specifies the test activities that must be repeated when testing is resumed.

7.1 Automated Unit Test Suite

As components are being developed, unit tests will be developed to test the interfaces of the components and low-level unit tests will be developed to test the methods of the underlying classes in the components.

As a prerequisite to the BAT, the automated unit test suite will be run by the build server on a per-build basis.

When the unit-test suite reports failures, testing will not occur on that build until the failures have been analyzed and resolved. Testing will resume on a build that passes the automated unit test suite.

7.2 Build Acceptance Test (BAT)

When a build is deemed ready to test by development, a build acceptance test will be run on the build. The BAT will consist of a broad but shallow set of tests to determine the overall stability of the build and decide if it is worth testing.

If the BAT fails on a particular build, testing will suspend until another build is created with any BAT failure issues fixed, verified by running the BAT again. Testing will resume on a build that passes the BAT.

Different build acceptance tests will be developed and used for the different test phases. Component BATs will be small and localized for each of the components. Integration BATs will vary based on the level of integration testing being performed. The System Test BAT will contain a set of tests that will utilize each of the components of the system.

7.3 Regression Testing

On a build by build basis, major bug fixes or code changes will be reviewed to determine the effects they may have on the system. If the changes are deemed to cause a sufficient amount of risk, regression test sets of the appropriately judged size will be created and executed.

A system-wide regression will also be run on the release candidate build to ensure incremental changes to the system have not altered the results of the tests that were run early in the test cycle.

7.4 System Design Changes

If at any point in time issues are submitted that require a design change to the system, all testing will be suspended. After the changes to the requirements, system architecture, and object design are made, a review and updates will be performed of the test specifications to ensure they properly align with the revised system changes. After updates are made, testing will resume. Tests in the vicinity of the change must all be rerun. A 20% regression of other tests must also be performed to ensure the changes did not adversely affect other parts of the system.
8. Testing materials (hardware/software requirements)
This section identifies the resources that are needed for testing. This should include the physical characteristics of the facilities, including the hardware, software, special test tools, and other resources needed (office space, etc.) to support the tests.

8.1 Facilities required

The team will need a lab are for the test equipment. The lab area should be approximately 400 sq ft in size, with 160 sq ft of desktop space. The lab area needs multiple power outlets on each side of the room. A table in the center of the room would also allow for easy-access test team technical discussions.

8.2 Hardware required

To enable the team to testing in an optimal environment, the lab needs to contain 4 copies of the system under test. The hardware components in the system [as seen in section 3.3 of the architecture specification], in a single set are a Database Server, a WebServer, a client PC with a web browser that supports Java, and the embedded system that is used in the ambulance. The four systems allow the team to test several components in parallel.

Also, if a single system exhibits a strange bug, it can be left in that state for developer debugging and analysis, while testing continues on the other systems.

8.2 Software required

Software required in this system is minimal. The database server needs the appropriate database (MySQL) installed, setup, and configured properly.

The WebServer machine needs IIS installed the the ISS services started so that it can properly act as a WebServer machine.

The client machine needs Java 1.5.0_b12 installed and properly configured with Internet Explorer 6.0 or newer.

8.3 Special requirements

Additional tools and software may need to be purchased or otherwise acquired or reused. Such software is used to execute special tests that are best run and results recorded and analyzed by automated means. For Load testing this requires a tool like LoadRunner. For Security testing at the complied source code level this requires a tool like FindBugs.
9. Test cases
This section, the core of the test plan, lists the test cases that are used during testing. Each test case is described in detail in a separate Test Case Specification document. Each execution of these tests will be documented in a Test Incident Report document.

9.1 Test specifications derived from nonfunctional requirements

nfrs_3.3.1_usability_MouseKeyboardNavigation
nfrs_3.3.1_usability_EaseOfUnderstandabilityAndUse
nfrs_3.3.1_usability_UserManual
nfrs_3.3.2_reliability_ErrorRecovery
nfrs_3.3.3_performance_DispatchTime3Minutes
nfrs_3.3.3_performance_11MinuteException
nfrs_3.3.3_performance_PageResponseTime15seconds
nfrs_3.3.6_interface_WebBrowserSupport

9.2 Test specifications derived from the use cases in the functional requirements

uc_3.4.2.2_1_Login
uc_3.4.2.2_1_LoginAlternateFlow
uc_3.4.2.2_2_CreateIncident
uc_3.4.2.2_2_CreateIncidentAlternateFlow
uc_3.4.2.2_3_FindIncidentLocation
uc_3.4.2.2_3_FindIncidentLocationAlternateFlow
uc_3.4.2.2_4_LocateAmbulance
uc_3.4.2.2_4_LocateAmbulanceAlternateFlow
uc_3.4.2.2_5_AllocateAmbulance
uc_3.4.2.2_6_AmbulanceTracking
uc_3.4.2.2_7_GetReports
uc_3.4.2.2_8_GeographicAnalysis
uc_3.4.2.2_9_ManageUsers
uc_3.4.2.2_10_DivertToPrivateParty
uc_3.4.5_UINavigation

9.3 Test specifications derived from the system architecture specification

arch_3.3_db_NetworkErr_Login
arch_3.3_db_NetworkErr_CreateIncident
arch_3.3_db_NetworkErr_LocateAmbulance
arch_3.3_db_NetworkErr_Dispatch
arch_3.3_db_NetworkErr_AmbulanceTracking
arch_3.3_db_NetworkErr_GetReports
arch_3.3_db_NetworkErr_ManageUsers

9.4 Test specifications derived from the subsystem decomposition and component diagram

odd_4.3_UserManagement_ValidateUser
odd_4.3_UserManagement_CreateUser
odd_4.3_UserManagement_UpdateUserInfo
odd_4.3_UserManagement_CreateReports
odd_4.4_DispatcherSystem_GetAddress
odd_4.4_DispatcherSystem_GetCoordinates
odd_4.4_DispatcherSystem_GetIncidentDetails
odd_4.4_DispatcherSystem_CreateNewIncident
odd_4.4_DispatcherSystem_LocateAmbulance
odd_4.4_DispatcherSystem_GetNearestHospital
odd_4.4_DispatcherSystem_AllocateAmbulance
odd_4.5_TrackingSystem_TrackAmbulance
odd_4.5_TrackingSystem_CreateMap
odd_4.5_TrackingSystem_SetAmbulanceStatus
odd_4.5_TrackingSystem_UpdateAmbulanceStatus
odd_4.5_TrackingSystem_MonitorTime
odd_4.6_GPSInterface_ProvideRouteDetails
odd_4.6_GPSInterface_ProvideCoordinates
odd_4.6_GPSInterface_CreateMap
odd_4.6_GPSInterface_ProvideStatus
odd_4.7_AddrInterface_FindAddress

9.5 Test case specifications that cover end-to-end test scenarios derived from the use cases

e2e_DispatchAndTrackSingleAmbulanceWhenAvailable
e2e_DispatchAndTrackMultipleAmbulancesWhenAvailable
e2e_DispatchAndTrackSingleAmbulanceWhenBusy
e2e_DispatchAndTrackMultipleAmbulancesWhenBusy
e2e_RequireManualIntervention
e2e_ConcurrentIncidents

10. Testing schedule
This section of the test plan covers responsibilities, staffing and training needs, risks and contingencies, and the test schedule.

10.1 Test Schedule
Test Phase Time Owner
Test Plan Creation 1 wk Test Manager
Test Specification Creation 2 wks Test Leads
Test Specification Team Review 1 wk Project Team
Component Testing 4 wks Component Testers
Integration Testing 4 wks Component & System Testers
System Testing 3 wks System Testers
Performance Testing 1 wk System Testers
Use Case Validation 2 wks System Testers
Alpha Testing 1 wk Product Managers / Analysts
Beta Testing Pilot Program 4 wks Pilot Customers
10.2 Responsibilities

The Test Manager is responsible for the overall test plan (this document) and test resources throughout the course of the project. He needs to be assigned to the project to review the requirements analysis, system architecture design, and object design of the system. From those specifications, he will generate the Test Plan. He will also generate section 9 “Test Cases” in this document. He will generate the list of test specifications and a brief description of each one in this document. He will generate and communicate the test strategy for the project to the test team and the rest of the project team as well as locate, acquire, and/or allocate the proper resources. He will provide periodic updates to the Program Director on the progress of test execution versus the plan as well as the metrics on the quality status of the product, focusing on key issues that need immediate attention from the Project Office.

The Test Leads are responsible for the creation of the detailed test specifications and will generate those and revise section 9 of this document as needed. The leads manage the day-to-day progress of each of their subcomponents and compile and report the metrics to the test manager. They are also responsible for ensuring the testers make adequate progress and follow the overall strategy defined by the Test Manager.

The Component Testers are responsible for the test execution on a daily basis for the component of the system to which they’ve been assigned. They also lead the effort during most of the integration test cycle and hand off the testing to the System Testers during the last states of integration testing.

The System Testers are responsible for functional testing, performance testing, and use case validation testing during the System Test Phase of the project.

During Alpha Testing, the Product Managers / Business Analysts who wrote the requirements will test the system to determine if the system truly aligns with the original vision of the system.

During Beta Testing, a small group of specially chosen customers will pilot the product to ensure it suits their needs.
10.3 Staffing and training needs

1 Test Manager
– responsible for finding and training the following test resources

3 Test Leads
– must be trained on the process being used for this project
– must be trained on the test specification format utilized
– must be trained on the defect/issue tracking system utilized

7 Component Testers
– must know SQL and Java
– must be familiar with NetBeans and Eclipse
– must be skilled at unit testing, API testing, and integration testing
– must be trained on the process being used for this project, the test specification format utilized, and the defect/issue tracking system utilized

7 System Testers
– must know how to use Load testing & Stress testing automation tools (such as LoadRunner)
– must be experienced in system testing and use case validation testing
– must be trained on the process being used for this project.
– must be trained on the test specification format utilized
– must be trained on the defect/issue tracking system utilized
10.4 Risks and contingencies matrix

Risk
Probability
Risk Type
Owner
Contingencies / Mitigation Approach
Unable to acquire the necessary number of skilled personnel as the components become ready to test.
30%
Personnel
Schedule
Test Manager
Resources for components will be split between the existing resources.
Schedule must be adjusted accordingly.
Unable to acquire some of the necessary hardware and software required for integration and system testing
25%
Equipment
Program Manager
Test Manager
Development Manager
Utilize existing acquired hardware.
Split test execution into morning and evening shifts such that testing can occur for multiple teams in the same day using the limited hardware. This requires support of the development during both shifts.
Third party services utilized in the system become unavailable during testing
5%
Third party
Alliance Manager
Setup a communication channel to 3rd party to report and handle issues when they occur.
Use the communication channel above to stay aware of planned outages and maintenance to help schedule test execution.
Components are not delivered on time
25%
Schedule
Development Manager
Integration testing with those components must be delayed until the component is delivered
Overall integration test approach may be modified to do an appropriate amount of bottom-up as well as top-down or sandwich integration.
Schedule must be adjusted accordingly.
Turnover
5%
Personnel
Test Manager
Testers will work in pairs on components. If a single member of the team decides to leave, a secondary testing with the knowledge of the component will still be able to train a new tester or finish the work.
Schedule must be adjusted accordingly.

2007.07.16

Scot Robinson

Initial document creation

0.2

Software Project Test Plan
Project: ADS – Ambulance Dispatch System
Purpose
The purpose of this document is to outline the test strategy and overall test approach for the ADS project. This includes test methodologies, traceability, resources required, and estimated schedule.

Audience
The audience of this document is the project team and the project management team. This document is also written for the extended test team. The test lead, testers, and any outsourced testers should be able to utilize this document to understand the scope of work that must be accomplished by the test team. The document is intended to accomplish its purpose only for the intended audiences.
Revision History
Revision
Date
Updated By
Update Comments
0.1
2007.07.16
Yo
Initial document creation
0.2
2007.07.20
Michael Jones
First Draft
0.3
2007.07.21
Arithmetic Ceramic
Ready for Review
0.4
2007.07.22
Final Revisions
1. Introduction
This section describes the objectives and extent of the tests. The goal is to provide a framework that can be used by managers and testers to plan and execute the necessary tests in a timely and cost-effective manner.

1.1 Test objectives
The objective of the test suite is to provide adequate coverage metrics, requirements validation, and system quality data such that sufficient data is provided for those making the decision to release.

1.2 Extent of tests
The tests referenced herein are written to validate use cases, requirements (both functional and non-functional), system architecture, and object design. The structured tests for object design will be run first as the components of the system are developed. The structured tests to validate the system architecture will be run next as the system is integrated in bottom-up fashion during integration test.
2. Relationship to other documents
This section explains the relationship of the test plan to the other documents produced during the development effort such as the RAD, SDD, and ODD (Object Design Document). It explains how all the tests are related to the functional and nonfunctional requirements, as well as to the system design stated in the respective documents. If necessary, this section introduces a naming scheme to establish the correspondence between requirements and tests.

2.1 Relationships to documents

Black box tests relating to use cases are developed from the use case diagram(s) in the RAD (requirements analysis document).

Black box tests derived from functional requirements are developed from the requirements lists in the RAD.

Performance tests derived from nonfunctional requirements are developed from the nonfunctional requirements in the RAD.

Structured (unit/white box) tests are generated from the OOD (Object Design Document). The specific tests are developed from the OOD component diagram of each of the components.

Integration tests are developed from the SDD (System/Architecture Design Document). The integration tests generally come from the overall package diagram describing the architecture of the system. The architecture is also used to help in determining the integration test approach. The test environment (hardware/software) is also derived from the SDD.

A visualization of the relationships to the other documents can be seen in the diagram below.
2.2 Test naming scheme

The names of test cases will indicate from where they have been derived using a system of prefixes. The following prefixes are use to denote the tests were derived from the following places.

uc_ tests derived from use cases
nfrs_ tests derived from nonfunctional requirements
arch_ integration tests derived from the system architecture spec
odd_ structured tests derived from the subsystem decomposition and component diagrams
e2e_ end-to-end systemic tests that exercise the entire user scenarios

If a test was derived from a particular requirement number or component design number, the test names will contain the requirement number or name after prefix followed by an underscore.

After the prefix and the number or name identifier, the test name shall contain a brief, but descriptive name.
3. System overview
This section, focusing on the structural aspects of testing, provides an overview of the system in terms of the components that are tested during the unit test. The granularity of components and their dependencies are defined in this section.

3.1 Software Architecture Overview

* Architecture Overview an excerpt from Deliverable 3: Architecture Specification
3.2 Subsystem Decomposition (Components)

* Subsystem Decomposition an excerpt from Deliverable 3: Architecture Specification

4. Features to be tested/not to be tested
This section, focusing on the functional aspects of testing, identifies all features and combinations of features to be tested. It also describes all those features that are not to be tested and the reasons for not testing them.

4.1 Features to be tested

Components developed in house

The components developed by this organization will be tested unless otherwise noted below in section 4.2.

Components developed by outsource vendors

Components outsourced to be developed specifically for this project where this test team has primary responsibility to test and validate those components will be tested.

Components outsourced to be developed specifically for this project where the outsourced vendor is responsible for development as well as testing will not be tested as components by the component testers. Rather, the test results from the vendor will be reviewed by the test leads, and if they pass review, the component will be tested in-house starting with integration test.
4.2 Items that will not be tested

3rd party and Off-The-Shelf components

It is assumed that 3rd party components were evaluated and the pros and cons properly weighed before choosing that component with our software. The interfaces to those components will be tested, but not the functionality or performance those components.

This includes any 3rd party websites and GPS devices or software.

Infrastructure components

The actual database software utilized is assumed to work as designed and will not be directly tested for functionality. Performance tests will be done during system test with respect to GUI response time that will involve the database. However, no testing will be done directly against the database.

The internet/WIFI backbone will be utilized during testing, however, no tests will be written executed to directly test the communications backbone.
5. Pass/Fail criteria
This section specifies generic pass/fail criteria for the tests covered in this plan. They are supplemented by pass/fail criteria in the test design specification. Note that “fail” in the IEEE standard terminology means “successful test” in our terminology.

5.1 Component Pass/Fail criteria

Tests executed on components only pass when they satisfy the signatures, constraints, and interfaces dictated by the Object Design Specification for that component. This includes positive tests, negative and stress tests, and boundary tests.

If a test exhibits a product failure to meet the objectives of the object design specification, it will fail and a defect/issue will be reported in the defect tracking system for review by the triage team.
5.2 Integration Pass/Fail criteria

Tests executed on integrated components only pass when they satisfy the signatures, constraints, and interfaces dictated by both the object design specification and the system architecture specification. This includes positive tests, negative and stress tests, boundary conditions, and tests that explicitly manipulate the interface environment (such as the physical connection to the database server).

If a test exhibits a product failure to meet the objectives of both the object design specification and the system architecture specification, it will fail and a defect/issue will be reported in the defect tracking system for review by the triage team.
5.3 System Pass/Fail criteria

Tests executed against the system use the functional requirements, non-functional requirements, and use cases as the oracle to determine pass or fail.

If a test exhibits a product failure to meet the objectives of any of the functional requirements, non-functional requirements, or the use cases, it will fail and a defect/issue will be reported in the defect tracking system for review by the triage team.
6. Approach
This section describes the general approach to the testing process. It discusses the reasons for the selected integration testing strategy. Different strategies are often needed to test different parts of the system.

6.1 General Test Strategy

Unit testing and component testing will be performed on the components as they are developed. Test will be executed using test code in the form of either custom test tools or as an automated suite of tests run against the components in their individual sandboxes.

Integrations tests will be performed by both the component testers as well as the system testers. The BAT and the unit test suite will be used as a regression during the integration of components. However, as the integration begins to include GUI level functionality, the tests being run will utilize significantly more manual testing and less automated testing.

System test will require a new set of tools that can measure NFRS compliance, such as LoadRunner (load testing) or FindBugs (java source code analysis for security issues). Manual tests will start by validating functionality based on the requirements. Later stages of system test will include manual end-to-end tests to validate use cases.

6.2 Integration Test Strategy

Because the components will be developed from the bottom-up and top-down, the test strategy will also align to the order of development of components. This will utilize a mostly bottom-up integration test approach, but will also involve the sandwich integration test approach.

Please review the system architecture overview and the subsystem architecture in section 3.

6.3 Test Case alignment with test phases

7. Suspension and resumption
This section specifies the criteria for suspending the testing on the test items associated with the plan. It also specifies the test activities that must be repeated when testing is resumed.

7.1 Automated Unit Test Suite

As components are being developed, unit tests will be developed to test the interfaces of the components and low-level unit tests will be developed to test the methods of the underlying classes in the components.

As a prerequisite to the BAT, the automated unit test suite will be run by the build server on a per-build basis.

When the unit-test suite reports failures, testing will not occur on that build until the failures have been analyzed and resolved. Testing will resume on a build that passes the automated unit test suite.

7.2 Build Acceptance Test (BAT)

When a build is deemed ready to test by development, a build acceptance test will be run on the build. The BAT will consist of a broad but shallow set of tests to determine the overall stability of the build and decide if it is worth testing.

If the BAT fails on a particular build, testing will suspend until another build is created with any BAT failure issues fixed, verified by running the BAT again. Testing will resume on a build that passes the BAT.

Different build acceptance tests will be developed and used for the different test phases. Component BATs will be small and localized for each of the components. Integration BATs will vary based on the level of integration testing being performed. The System Test BAT will contain a set of tests that will utilize each of the components of the system.

7.3 Regression Testing

On a build by build basis, major bug fixes or code changes will be reviewed to determine the effects they may have on the system. If the changes are deemed to cause a sufficient amount of risk, regression test sets of the appropriately judged size will be created and executed.

A system-wide regression will also be run on the release candidate build to ensure incremental changes to the system have not altered the results of the tests that were run early in the test cycle.

7.4 System Design Changes

If at any point in time issues are submitted that require a design change to the system, all testing will be suspended. After the changes to the requirements, system architecture, and object design are made, a review and updates will be performed of the test specifications to ensure they properly align with the revised system changes. After updates are made, testing will resume. Tests in the vicinity of the change must all be rerun. A 20% regression of other tests must also be performed to ensure the changes did not adversely affect other parts of the system.
8. Testing materials (hardware/software requirements)
This section identifies the resources that are needed for testing. This should include the physical characteristics of the facilities, including the hardware, software, special test tools, and other resources needed (office space, etc.) to support the tests.

8.1 Facilities required

The team will need a lab are for the test equipment. The lab area should be approximately 400 sq ft in size, with 160 sq ft of desktop space. The lab area needs multiple power outlets on each side of the room. A table in the center of the room would also allow for easy-access test team technical discussions.

8.2 Hardware required

To enable the team to testing in an optimal environment, the lab needs to contain 4 copies of the system under test. The hardware components in the system [as seen in section 3.3 of the architecture specification], in a single set are a Database Server, a WebServer, a client PC with a web browser that supports Java, and the embedded system that is used in the ambulance. The four systems allow the team to test several components in parallel.

Also, if a single system exhibits a strange bug, it can be left in that state for developer debugging and analysis, while testing continues on the other systems.

8.2 Software required

Software required in this system is minimal. The database server needs the appropriate database (MySQL) installed, setup, and configured properly.

The WebServer machine needs IIS installed the the ISS services started so that it can properly act as a WebServer machine.

The client machine needs Java 1.5.0_b12 installed and properly configured with Internet Explorer 6.0 or newer.

8.3 Special requirements

Additional tools and software may need to be purchased or otherwise acquired or reused. Such software is used to execute special tests that are best run and results recorded and analyzed by automated means. For Load testing this requires a tool like LoadRunner. For Security testing at the complied source code level this requires a tool like FindBugs.
9. Test cases
This section, the core of the test plan, lists the test cases that are used during testing. Each test case is described in detail in a separate Test Case Specification document. Each execution of these tests will be documented in a Test Incident Report document.

9.1 Test specifications derived from nonfunctional requirements

nfrs_3.3.1_usability_MouseKeyboardNavigation
nfrs_3.3.1_usability_EaseOfUnderstandabilityAndUse
nfrs_3.3.1_usability_UserManual
nfrs_3.3.2_reliability_ErrorRecovery
nfrs_3.3.3_performance_DispatchTime3Minutes
nfrs_3.3.3_performance_11MinuteException
nfrs_3.3.3_performance_PageResponseTime15seconds
nfrs_3.3.6_interface_WebBrowserSupport

9.2 Test specifications derived from the use cases in the functional requirements

uc_3.4.2.2_1_Login
uc_3.4.2.2_1_LoginAlternateFlow
uc_3.4.2.2_2_CreateIncident
uc_3.4.2.2_2_CreateIncidentAlternateFlow
uc_3.4.2.2_3_FindIncidentLocation
uc_3.4.2.2_3_FindIncidentLocationAlternateFlow
uc_3.4.2.2_4_LocateAmbulance
uc_3.4.2.2_4_LocateAmbulanceAlternateFlow
uc_3.4.2.2_5_AllocateAmbulance
uc_3.4.2.2_6_AmbulanceTracking
uc_3.4.2.2_7_GetReports
uc_3.4.2.2_8_GeographicAnalysis
uc_3.4.2.2_9_ManageUsers
uc_3.4.2.2_10_DivertToPrivateParty
uc_3.4.5_UINavigation

9.3 Test specifications derived from the system architecture specification

arch_3.3_db_NetworkErr_Login
arch_3.3_db_NetworkErr_CreateIncident
arch_3.3_db_NetworkErr_LocateAmbulance
arch_3.3_db_NetworkErr_Dispatch
arch_3.3_db_NetworkErr_AmbulanceTracking
arch_3.3_db_NetworkErr_GetReports
arch_3.3_db_NetworkErr_ManageUsers

9.4 Test specifications derived from the subsystem decomposition and component diagram

odd_4.3_UserManagement_ValidateUser
odd_4.3_UserManagement_CreateUser
odd_4.3_UserManagement_UpdateUserInfo
odd_4.3_UserManagement_CreateReports
odd_4.4_DispatcherSystem_GetAddress
odd_4.4_DispatcherSystem_GetCoordinates
odd_4.4_DispatcherSystem_GetIncidentDetails
odd_4.4_DispatcherSystem_CreateNewIncident
odd_4.4_DispatcherSystem_LocateAmbulance
odd_4.4_DispatcherSystem_GetNearestHospital
odd_4.4_DispatcherSystem_AllocateAmbulance
odd_4.5_TrackingSystem_TrackAmbulance
odd_4.5_TrackingSystem_CreateMap
odd_4.5_TrackingSystem_SetAmbulanceStatus
odd_4.5_TrackingSystem_UpdateAmbulanceStatus
odd_4.5_TrackingSystem_MonitorTime
odd_4.6_GPSInterface_ProvideRouteDetails
odd_4.6_GPSInterface_ProvideCoordinates
odd_4.6_GPSInterface_CreateMap
odd_4.6_GPSInterface_ProvideStatus
odd_4.7_AddrInterface_FindAddress

9.5 Test case specifications that cover end-to-end test scenarios derived from the use cases

e2e_DispatchAndTrackSingleAmbulanceWhenAvailable
e2e_DispatchAndTrackMultipleAmbulancesWhenAvailable
e2e_DispatchAndTrackSingleAmbulanceWhenBusy
e2e_DispatchAndTrackMultipleAmbulancesWhenBusy
e2e_RequireManualIntervention
e2e_ConcurrentIncidents

10. Testing schedule
This section of the test plan covers responsibilities, staffing and training needs, risks and contingencies, and the test schedule.

10.1 Test Schedule
Test Phase Time Owner
Test Plan Creation 1 wk Test Manager
Test Specification Creation 2 wks Test Leads
Test Specification Team Review 1 wk Project Team
Component Testing 4 wks Component Testers
Integration Testing 4 wks Component & System Testers
System Testing 3 wks System Testers
Performance Testing 1 wk System Testers
Use Case Validation 2 wks System Testers
Alpha Testing 1 wk Product Managers / Analysts
Beta Testing Pilot Program 4 wks Pilot Customers
10.2 Responsibilities

The Test Manager is responsible for the overall test plan (this document) and test resources throughout the course of the project. He needs to be assigned to the project to review the requirements analysis, system architecture design, and object design of the system. From those specifications, he will generate the Test Plan. He will also generate section 9 “Test Cases” in this document. He will generate the list of test specifications and a brief description of each one in this document. He will generate and communicate the test strategy for the project to the test team and the rest of the project team as well as locate, acquire, and/or allocate the proper resources. He will provide periodic updates to the Program Director on the progress of test execution versus the plan as well as the metrics on the quality status of the product, focusing on key issues that need immediate attention from the Project Office.

The Test Leads are responsible for the creation of the detailed test specifications and will generate those and revise section 9 of this document as needed. The leads manage the day-to-day progress of each of their subcomponents and compile and report the metrics to the test manager. They are also responsible for ensuring the testers make adequate progress and follow the overall strategy defined by the Test Manager.

The Component Testers are responsible for the test execution on a daily basis for the component of the system to which they’ve been assigned. They also lead the effort during most of the integration test cycle and hand off the testing to the System Testers during the last states of integration testing.

The System Testers are responsible for functional testing, performance testing, and use case validation testing during the System Test Phase of the project.

During Alpha Testing, the Product Managers / Business Analysts who wrote the requirements will test the system to determine if the system truly aligns with the original vision of the system.

During Beta Testing, a small group of specially chosen customers will pilot the product to ensure it suits their needs.
10.3 Staffing and training needs

1 Test Manager
– responsible for finding and training the following test resources

3 Test Leads
– must be trained on the process being used for this project
– must be trained on the test specification format utilized
– must be trained on the defect/issue tracking system utilized

7 Component Testers
– must know SQL and Java
– must be familiar with NetBeans and Eclipse
– must be skilled at unit testing, API testing, and integration testing
– must be trained on the process being used for this project, the test specification format utilized, and the defect/issue tracking system utilized

7 System Testers
– must know how to use Load testing & Stress testing automation tools (such as LoadRunner)
– must be experienced in system testing and use case validation testing
– must be trained on the process being used for this project.
– must be trained on the test specification format utilized
– must be trained on the defect/issue tracking system utilized
10.4 Risks and contingencies matrix

Risk
Probability
Risk Type
Owner
Contingencies / Mitigation Approach
Unable to acquire the necessary number of skilled personnel as the components become ready to test.
30%
Personnel
Schedule
Test Manager
Resources for components will be split between the existing resources.
Schedule must be adjusted accordingly.
Unable to acquire some of the necessary hardware and software required for integration and system testing
25%
Equipment
Program Manager
Test Manager
Development Manager
Utilize existing acquired hardware.
Split test execution into morning and evening shifts such that testing can occur for multiple teams in the same day using the limited hardware. This requires support of the development during both shifts.
Third party services utilized in the system become unavailable during testing
5%
Third party
Alliance Manager
Setup a communication channel to 3rd party to report and handle issues when they occur.
Use the communication channel above to stay aware of planned outages and maintenance to help schedule test execution.
Components are not delivered on time
25%
Schedule
Development Manager
Integration testing with those components must be delayed until the component is delivered
Overall integration test approach may be modified to do an appropriate amount of bottom-up as well as top-down or sandwich integration.
Schedule must be adjusted accordingly.
Turnover
5%
Personnel
Test Manager
Testers will work in pairs on components. If a single member of the team decides to leave, a secondary testing with the knowledge of the component will still be able to train a new tester or finish the work.
Schedule must be adjusted accordingly.

2007.07.20

Scot Robinson

First Draft

0.3

Software Project Test Plan
Project: ADS – Ambulance Dispatch System
Purpose
The purpose of this document is to outline the test strategy and overall test approach for the ADS project. This includes test methodologies, traceability, resources required, and estimated schedule.

Audience
The audience of this document is the project team and the project management team. This document is also written for the extended test team. The test lead, testers, and any outsourced testers should be able to utilize this document to understand the scope of work that must be accomplished by the test team. The document is intended to accomplish its purpose only for the intended audiences.
Revision History
Revision
Date
Updated By
Update Comments
0.1
2007.07.16
Michael Jones
Initial document creation
0.2
2007.07.20
First Draft
0.3
2007.07.21
Ready for Review
0.4
2007.07.22
Final Revisions
1. Introduction
This section describes the objectives and extent of the tests. The goal is to provide a framework that can be used by managers and testers to plan and execute the necessary tests in a timely and cost-effective manner.

1.1 Test objectives
The objective of the test suite is to provide adequate coverage metrics, requirements validation, and system quality data such that sufficient data is provided for those making the decision to release.

1.2 Extent of tests
The tests referenced herein are written to validate use cases, requirements (both functional and non-functional), system architecture, and object design. The structured tests for object design will be run first as the components of the system are developed. The structured tests to validate the system architecture will be run next as the system is integrated in bottom-up fashion during integration test.
2. Relationship to other documents
This section explains the relationship of the test plan to the other documents produced during the development effort such as the RAD, SDD, and ODD (Object Design Document). It explains how all the tests are related to the functional and nonfunctional requirements, as well as to the system design stated in the respective documents. If necessary, this section introduces a naming scheme to establish the correspondence between requirements and tests.

2.1 Relationships to documents

Black box tests relating to use cases are developed from the use case diagram(s) in the RAD (requirements analysis document).

Black box tests derived from functional requirements are developed from the requirements lists in the RAD.

Performance tests derived from nonfunctional requirements are developed from the nonfunctional requirements in the RAD.

Structured (unit/white box) tests are generated from the OOD (Object Design Document). The specific tests are developed from the OOD component diagram of each of the components.

Integration tests are developed from the SDD (System/Architecture Design Document). The integration tests generally come from the overall package diagram describing the architecture of the system. The architecture is also used to help in determining the integration test approach. The test environment (hardware/software) is also derived from the SDD.

A visualization of the relationships to the other documents can be seen in the diagram below.
2.2 Test naming scheme

The names of test cases will indicate from where they have been derived using a system of prefixes. The following prefixes are use to denote the tests were derived from the following places.

uc_ tests derived from use cases
nfrs_ tests derived from nonfunctional requirements
arch_ integration tests derived from the system architecture spec
odd_ structured tests derived from the subsystem decomposition and component diagrams
e2e_ end-to-end systemic tests that exercise the entire user scenarios

If a test was derived from a particular requirement number or component design number, the test names will contain the requirement number or name after prefix followed by an underscore.

After the prefix and the number or name identifier, the test name shall contain a brief, but descriptive name.
3. System overview
This section, focusing on the structural aspects of testing, provides an overview of the system in terms of the components that are tested during the unit test. The granularity of components and their dependencies are defined in this section.

3.1 Software Architecture Overview

* Architecture Overview an excerpt from Deliverable 3: Architecture Specification
3.2 Subsystem Decomposition (Components)

* Subsystem Decomposition an excerpt from Deliverable 3: Architecture Specification

4. Features to be tested/not to be tested
This section, focusing on the functional aspects of testing, identifies all features and combinations of features to be tested. It also describes all those features that are not to be tested and the reasons for not testing them.

4.1 Features to be tested

Components developed in house

The components developed by this organization will be tested unless otherwise noted below in section 4.2.

Components developed by outsource vendors

Components outsourced to be developed specifically for this project where this test team has primary responsibility to test and validate those components will be tested.

Components outsourced to be developed specifically for this project where the outsourced vendor is responsible for development as well as testing will not be tested as components by the component testers. Rather, the test results from the vendor will be reviewed by the test leads, and if they pass review, the component will be tested in-house starting with integration test.
4.2 Items that will not be tested

3rd party and Off-The-Shelf components

It is assumed that 3rd party components were evaluated and the pros and cons properly weighed before choosing that component with our software. The interfaces to those components will be tested, but not the functionality or performance those components.

This includes any 3rd party websites and GPS devices or software.

Infrastructure components

The actual database software utilized is assumed to work as designed and will not be directly tested for functionality. Performance tests will be done during system test with respect to GUI response time that will involve the database. However, no testing will be done directly against the database.

The internet/WIFI backbone will be utilized during testing, however, no tests will be written executed to directly test the communications backbone.
5. Pass/Fail criteria
This section specifies generic pass/fail criteria for the tests covered in this plan. They are supplemented by pass/fail criteria in the test design specification. Note that “fail” in the IEEE standard terminology means “successful test” in our terminology.

5.1 Component Pass/Fail criteria

Tests executed on components only pass when they satisfy the signatures, constraints, and interfaces dictated by the Object Design Specification for that component. This includes positive tests, negative and stress tests, and boundary tests.

If a test exhibits a product failure to meet the objectives of the object design specification, it will fail and a defect/issue will be reported in the defect tracking system for review by the triage team.
5.2 Integration Pass/Fail criteria

Tests executed on integrated components only pass when they satisfy the signatures, constraints, and interfaces dictated by both the object design specification and the system architecture specification. This includes positive tests, negative and stress tests, boundary conditions, and tests that explicitly manipulate the interface environment (such as the physical connection to the database server).

If a test exhibits a product failure to meet the objectives of both the object design specification and the system architecture specification, it will fail and a defect/issue will be reported in the defect tracking system for review by the triage team.
5.3 System Pass/Fail criteria

Tests executed against the system use the functional requirements, non-functional requirements, and use cases as the oracle to determine pass or fail.

If a test exhibits a product failure to meet the objectives of any of the functional requirements, non-functional requirements, or the use cases, it will fail and a defect/issue will be reported in the defect tracking system for review by the triage team.
6. Approach
This section describes the general approach to the testing process. It discusses the reasons for the selected integration testing strategy. Different strategies are often needed to test different parts of the system.

6.1 General Test Strategy

Unit testing and component testing will be performed on the components as they are developed. Test will be executed using test code in the form of either custom test tools or as an automated suite of tests run against the components in their individual sandboxes.

Integrations tests will be performed by both the component testers as well as the system testers. The BAT and the unit test suite will be used as a regression during the integration of components. However, as the integration begins to include GUI level functionality, the tests being run will utilize significantly more manual testing and less automated testing.

System test will require a new set of tools that can measure NFRS compliance, such as LoadRunner (load testing) or FindBugs (java source code analysis for security issues). Manual tests will start by validating functionality based on the requirements. Later stages of system test will include manual end-to-end tests to validate use cases.

6.2 Integration Test Strategy

Because the components will be developed from the bottom-up and top-down, the test strategy will also align to the order of development of components. This will utilize a mostly bottom-up integration test approach, but will also involve the sandwich integration test approach.

Please review the system architecture overview and the subsystem architecture in section 3.

6.3 Test Case alignment with test phases

7. Suspension and resumption
This section specifies the criteria for suspending the testing on the test items associated with the plan. It also specifies the test activities that must be repeated when testing is resumed.

7.1 Automated Unit Test Suite

As components are being developed, unit tests will be developed to test the interfaces of the components and low-level unit tests will be developed to test the methods of the underlying classes in the components.

As a prerequisite to the BAT, the automated unit test suite will be run by the build server on a per-build basis.

When the unit-test suite reports failures, testing will not occur on that build until the failures have been analyzed and resolved. Testing will resume on a build that passes the automated unit test suite.

7.2 Build Acceptance Test (BAT)

When a build is deemed ready to test by development, a build acceptance test will be run on the build. The BAT will consist of a broad but shallow set of tests to determine the overall stability of the build and decide if it is worth testing.

If the BAT fails on a particular build, testing will suspend until another build is created with any BAT failure issues fixed, verified by running the BAT again. Testing will resume on a build that passes the BAT.

Different build acceptance tests will be developed and used for the different test phases. Component BATs will be small and localized for each of the components. Integration BATs will vary based on the level of integration testing being performed. The System Test BAT will contain a set of tests that will utilize each of the components of the system.

7.3 Regression Testing

On a build by build basis, major bug fixes or code changes will be reviewed to determine the effects they may have on the system. If the changes are deemed to cause a sufficient amount of risk, regression test sets of the appropriately judged size will be created and executed.

A system-wide regression will also be run on the release candidate build to ensure incremental changes to the system have not altered the results of the tests that were run early in the test cycle.

7.4 System Design Changes

If at any point in time issues are submitted that require a design change to the system, all testing will be suspended. After the changes to the requirements, system architecture, and object design are made, a review and updates will be performed of the test specifications to ensure they properly align with the revised system changes. After updates are made, testing will resume. Tests in the vicinity of the change must all be rerun. A 20% regression of other tests must also be performed to ensure the changes did not adversely affect other parts of the system.
8. Testing materials (hardware/software requirements)
This section identifies the resources that are needed for testing. This should include the physical characteristics of the facilities, including the hardware, software, special test tools, and other resources needed (office space, etc.) to support the tests.

8.1 Facilities required

The team will need a lab are for the test equipment. The lab area should be approximately 400 sq ft in size, with 160 sq ft of desktop space. The lab area needs multiple power outlets on each side of the room. A table in the center of the room would also allow for easy-access test team technical discussions.

8.2 Hardware required

To enable the team to testing in an optimal environment, the lab needs to contain 4 copies of the system under test. The hardware components in the system [as seen in section 3.3 of the architecture specification], in a single set are a Database Server, a WebServer, a client PC with a web browser that supports Java, and the embedded system that is used in the ambulance. The four systems allow the team to test several components in parallel.

Also, if a single system exhibits a strange bug, it can be left in that state for developer debugging and analysis, while testing continues on the other systems.

8.2 Software required

Software required in this system is minimal. The database server needs the appropriate database (MySQL) installed, setup, and configured properly.

The WebServer machine needs IIS installed the the ISS services started so that it can properly act as a WebServer machine.

The client machine needs Java 1.5.0_b12 installed and properly configured with Internet Explorer 6.0 or newer.

8.3 Special requirements

Additional tools and software may need to be purchased or otherwise acquired or reused. Such software is used to execute special tests that are best run and results recorded and analyzed by automated means. For Load testing this requires a tool like LoadRunner. For Security testing at the complied source code level this requires a tool like FindBugs.
9. Test cases
This section, the core of the test plan, lists the test cases that are used during testing. Each test case is described in detail in a separate Test Case Specification document. Each execution of these tests will be documented in a Test Incident Report document.

9.1 Test specifications derived from nonfunctional requirements

nfrs_3.3.1_usability_MouseKeyboardNavigation
nfrs_3.3.1_usability_EaseOfUnderstandabilityAndUse
nfrs_3.3.1_usability_UserManual
nfrs_3.3.2_reliability_ErrorRecovery
nfrs_3.3.3_performance_DispatchTime3Minutes
nfrs_3.3.3_performance_11MinuteException
nfrs_3.3.3_performance_PageResponseTime15seconds
nfrs_3.3.6_interface_WebBrowserSupport

9.2 Test specifications derived from the use cases in the functional requirements

uc_3.4.2.2_1_Login
uc_3.4.2.2_1_LoginAlternateFlow
uc_3.4.2.2_2_CreateIncident
uc_3.4.2.2_2_CreateIncidentAlternateFlow
uc_3.4.2.2_3_FindIncidentLocation
uc_3.4.2.2_3_FindIncidentLocationAlternateFlow
uc_3.4.2.2_4_LocateAmbulance
uc_3.4.2.2_4_LocateAmbulanceAlternateFlow
uc_3.4.2.2_5_AllocateAmbulance
uc_3.4.2.2_6_AmbulanceTracking
uc_3.4.2.2_7_GetReports
uc_3.4.2.2_8_GeographicAnalysis
uc_3.4.2.2_9_ManageUsers
uc_3.4.2.2_10_DivertToPrivateParty
uc_3.4.5_UINavigation

9.3 Test specifications derived from the system architecture specification

arch_3.3_db_NetworkErr_Login
arch_3.3_db_NetworkErr_CreateIncident
arch_3.3_db_NetworkErr_LocateAmbulance
arch_3.3_db_NetworkErr_Dispatch
arch_3.3_db_NetworkErr_AmbulanceTracking
arch_3.3_db_NetworkErr_GetReports
arch_3.3_db_NetworkErr_ManageUsers

9.4 Test specifications derived from the subsystem decomposition and component diagram

odd_4.3_UserManagement_ValidateUser
odd_4.3_UserManagement_CreateUser
odd_4.3_UserManagement_UpdateUserInfo
odd_4.3_UserManagement_CreateReports
odd_4.4_DispatcherSystem_GetAddress
odd_4.4_DispatcherSystem_GetCoordinates
odd_4.4_DispatcherSystem_GetIncidentDetails
odd_4.4_DispatcherSystem_CreateNewIncident
odd_4.4_DispatcherSystem_LocateAmbulance
odd_4.4_DispatcherSystem_GetNearestHospital
odd_4.4_DispatcherSystem_AllocateAmbulance
odd_4.5_TrackingSystem_TrackAmbulance
odd_4.5_TrackingSystem_CreateMap
odd_4.5_TrackingSystem_SetAmbulanceStatus
odd_4.5_TrackingSystem_UpdateAmbulanceStatus
odd_4.5_TrackingSystem_MonitorTime
odd_4.6_GPSInterface_ProvideRouteDetails
odd_4.6_GPSInterface_ProvideCoordinates
odd_4.6_GPSInterface_CreateMap
odd_4.6_GPSInterface_ProvideStatus
odd_4.7_AddrInterface_FindAddress

9.5 Test case specifications that cover end-to-end test scenarios derived from the use cases

e2e_DispatchAndTrackSingleAmbulanceWhenAvailable
e2e_DispatchAndTrackMultipleAmbulancesWhenAvailable
e2e_DispatchAndTrackSingleAmbulanceWhenBusy
e2e_DispatchAndTrackMultipleAmbulancesWhenBusy
e2e_RequireManualIntervention
e2e_ConcurrentIncidents

10. Testing schedule
This section of the test plan covers responsibilities, staffing and training needs, risks and contingencies, and the test schedule.

10.1 Test Schedule
Test Phase Time Owner
Test Plan Creation 1 wk Test Manager
Test Specification Creation 2 wks Test Leads
Test Specification Team Review 1 wk Project Team
Component Testing 4 wks Component Testers
Integration Testing 4 wks Component & System Testers
System Testing 3 wks System Testers
Performance Testing 1 wk System Testers
Use Case Validation 2 wks System Testers
Alpha Testing 1 wk Product Managers / Analysts
Beta Testing Pilot Program 4 wks Pilot Customers
10.2 Responsibilities

The Test Manager is responsible for the overall test plan (this document) and test resources throughout the course of the project. He needs to be assigned to the project to review the requirements analysis, system architecture design, and object design of the system. From those specifications, he will generate the Test Plan. He will also generate section 9 “Test Cases” in this document. He will generate the list of test specifications and a brief description of each one in this document. He will generate and communicate the test strategy for the project to the test team and the rest of the project team as well as locate, acquire, and/or allocate the proper resources. He will provide periodic updates to the Program Director on the progress of test execution versus the plan as well as the metrics on the quality status of the product, focusing on key issues that need immediate attention from the Project Office.

The Test Leads are responsible for the creation of the detailed test specifications and will generate those and revise section 9 of this document as needed. The leads manage the day-to-day progress of each of their subcomponents and compile and report the metrics to the test manager. They are also responsible for ensuring the testers make adequate progress and follow the overall strategy defined by the Test Manager.

The Component Testers are responsible for the test execution on a daily basis for the component of the system to which they’ve been assigned. They also lead the effort during most of the integration test cycle and hand off the testing to the System Testers during the last states of integration testing.

The System Testers are responsible for functional testing, performance testing, and use case validation testing during the System Test Phase of the project.

During Alpha Testing, the Product Managers / Business Analysts who wrote the requirements will test the system to determine if the system truly aligns with the original vision of the system.

During Beta Testing, a small group of specially chosen customers will pilot the product to ensure it suits their needs.
10.3 Staffing and training needs

1 Test Manager
– responsible for finding and training the following test resources

3 Test Leads
– must be trained on the process being used for this project
– must be trained on the test specification format utilized
– must be trained on the defect/issue tracking system utilized

7 Component Testers
– must know SQL and Java
– must be familiar with NetBeans and Eclipse
– must be skilled at unit testing, API testing, and integration testing
– must be trained on the process being used for this project, the test specification format utilized, and the defect/issue tracking system utilized

7 System Testers
– must know how to use Load testing & Stress testing automation tools (such as LoadRunner)
– must be experienced in system testing and use case validation testing
– must be trained on the process being used for this project.
– must be trained on the test specification format utilized
– must be trained on the defect/issue tracking system utilized
10.4 Risks and contingencies matrix

Risk
Probability
Risk Type
Owner
Contingencies / Mitigation Approach
Unable to acquire the necessary number of skilled personnel as the components become ready to test.
30%
Personnel
Schedule
Test Manager
Resources for components will be split between the existing resources.
Schedule must be adjusted accordingly.
Unable to acquire some of the necessary hardware and software required for integration and system testing
25%
Equipment
Program Manager
Test Manager
Development Manager
Utilize existing acquired hardware.
Split test execution into morning and evening shifts such that testing can occur for multiple teams in the same day using the limited hardware. This requires support of the development during both shifts.
Third party services utilized in the system become unavailable during testing
5%
Third party
Alliance Manager
Setup a communication channel to 3rd party to report and handle issues when they occur.
Use the communication channel above to stay aware of planned outages and maintenance to help schedule test execution.
Components are not delivered on time
25%
Schedule
Development Manager
Integration testing with those components must be delayed until the component is delivered
Overall integration test approach may be modified to do an appropriate amount of bottom-up as well as top-down or sandwich integration.
Schedule must be adjusted accordingly.
Turnover
5%
Personnel
Test Manager
Testers will work in pairs on components. If a single member of the team decides to leave, a secondary testing with the knowledge of the component will still be able to train a new tester or finish the work.
Schedule must be adjusted accordingly.

2007.07.21

Scot Robinson

Ready for Review

0.4

Software Project Test Plan
Project: ADS – Ambulance Dispatch System
Purpose
The purpose of this document is to outline the test strategy and overall test approach for the ADS project. This includes test methodologies, traceability, resources required, and estimated schedule.

Audience
The audience of this document is the project team and the project management team. This document is also written for the extended test team. The test lead, testers, and any outsourced testers should be able to utilize this document to understand the scope of work that must be accomplished by the test team. The document is intended to accomplish its purpose only for the intended audiences.
Revision History
Revision
Date
Updated By
Update Comments
0.1
2007.07.16
Yo
Initial document creation
0.2
2007.07.20
Scot Miller
First Draft
0.3
2007.07.21
Rachel Shauna
Ready for Review
0.4
2007.07.22
Michael Jones
Final Revisions
1. Introduction
This section describes the objectives and extent of the tests. The goal is to provide a framework that can be used by managers and testers to plan and execute the necessary tests in a timely and cost-effective manner.

1.1 Test objectives
The objective of the test suite is to provide adequate coverage metrics, requirements validation, and system quality data such that sufficient data is provided for those making the decision to release.

1.2 Extent of tests
The tests referenced herein are written to validate use cases, requirements (both functional and non-functional), system architecture, and object design. The structured tests for object design will be run first as the components of the system are developed. The structured tests to validate the system architecture will be run next as the system is integrated in bottom-up fashion during integration test.
2. Relationship to other documents
This section explains the relationship of the test plan to the other documents produced during the development effort such as the RAD, SDD, and ODD (Object Design Document). It explains how all the tests are related to the functional and nonfunctional requirements, as well as to the system design stated in the respective documents. If necessary, this section introduces a naming scheme to establish the correspondence between requirements and tests.

2.1 Relationships to documents

Black box tests relating to use cases are developed from the use case diagram(s) in the RAD (requirements analysis document).

Black box tests derived from functional requirements are developed from the requirements lists in the RAD.

Performance tests derived from nonfunctional requirements are developed from the nonfunctional requirements in the RAD.

Structured (unit/white box) tests are generated from the OOD (Object Design Document). The specific tests are developed from the OOD component diagram of each of the components.

Integration tests are developed from the SDD (System/Architecture Design Document). The integration tests generally come from the overall package diagram describing the architecture of the system. The architecture is also used to help in determining the integration test approach. The test environment (hardware/software) is also derived from the SDD.

A visualization of the relationships to the other documents can be seen in the diagram below.
2.2 Test naming scheme

The names of test cases will indicate from where they have been derived using a system of prefixes. The following prefixes are use to denote the tests were derived from the following places.

uc_ tests derived from use cases
nfrs_ tests derived from nonfunctional requirements
arch_ integration tests derived from the system architecture spec
odd_ structured tests derived from the subsystem decomposition and component diagrams
e2e_ end-to-end systemic tests that exercise the entire user scenarios

If a test was derived from a particular requirement number or component design number, the test names will contain the requirement number or name after prefix followed by an underscore.

After the prefix and the number or name identifier, the test name shall contain a brief, but descriptive name.
3. System overview
This section, focusing on the structural aspects of testing, provides an overview of the system in terms of the components that are tested during the unit test. The granularity of components and their dependencies are defined in this section.

3.1 Software Architecture Overview

* Architecture Overview an excerpt from Deliverable 3: Architecture Specification
3.2 Subsystem Decomposition (Components)

* Subsystem Decomposition an excerpt from Deliverable 3: Architecture Specification

4. Features to be tested/not to be tested
This section, focusing on the functional aspects of testing, identifies all features and combinations of features to be tested. It also describes all those features that are not to be tested and the reasons for not testing them.

4.1 Features to be tested

Components developed in house

The components developed by this organization will be tested unless otherwise noted below in section 4.2.

Components developed by outsource vendors

Components outsourced to be developed specifically for this project where this test team has primary responsibility to test and validate those components will be tested.

Components outsourced to be developed specifically for this project where the outsourced vendor is responsible for development as well as testing will not be tested as components by the component testers. Rather, the test results from the vendor will be reviewed by the test leads, and if they pass review, the component will be tested in-house starting with integration test.
4.2 Items that will not be tested

3rd party and Off-The-Shelf components

It is assumed that 3rd party components were evaluated and the pros and cons properly weighed before choosing that component with our software. The interfaces to those components will be tested, but not the functionality or performance those components.

This includes any 3rd party websites and GPS devices or software.

Infrastructure components

The actual database software utilized is assumed to work as designed and will not be directly tested for functionality. Performance tests will be done during system test with respect to GUI response time that will involve the database. However, no testing will be done directly against the database.

The internet/WIFI backbone will be utilized during testing, however, no tests will be written executed to directly test the communications backbone.
5. Pass/Fail criteria
This section specifies generic pass/fail criteria for the tests covered in this plan. They are supplemented by pass/fail criteria in the test design specification. Note that “fail” in the IEEE standard terminology means “successful test” in our terminology.

5.1 Component Pass/Fail criteria

Tests executed on components only pass when they satisfy the signatures, constraints, and interfaces dictated by the Object Design Specification for that component. This includes positive tests, negative and stress tests, and boundary tests.

If a test exhibits a product failure to meet the objectives of the object design specification, it will fail and a defect/issue will be reported in the defect tracking system for review by the triage team.
5.2 Integration Pass/Fail criteria

Tests executed on integrated components only pass when they satisfy the signatures, constraints, and interfaces dictated by both the object design specification and the system architecture specification. This includes positive tests, negative and stress tests, boundary conditions, and tests that explicitly manipulate the interface environment (such as the physical connection to the database server).

If a test exhibits a product failure to meet the objectives of both the object design specification and the system architecture specification, it will fail and a defect/issue will be reported in the defect tracking system for review by the triage team.
5.3 System Pass/Fail criteria

Tests executed against the system use the functional requirements, non-functional requirements, and use cases as the oracle to determine pass or fail.

If a test exhibits a product failure to meet the objectives of any of the functional requirements, non-functional requirements, or the use cases, it will fail and a defect/issue will be reported in the defect tracking system for review by the triage team.
6. Approach
This section describes the general approach to the testing process. It discusses the reasons for the selected integration testing strategy. Different strategies are often needed to test different parts of the system.

6.1 General Test Strategy

Unit testing and component testing will be performed on the components as they are developed. Test will be executed using test code in the form of either custom test tools or as an automated suite of tests run against the components in their individual sandboxes.

Integrations tests will be performed by both the component testers as well as the system testers. The BAT and the unit test suite will be used as a regression during the integration of components. However, as the integration begins to include GUI level functionality, the tests being run will utilize significantly more manual testing and less automated testing.

System test will require a new set of tools that can measure NFRS compliance, such as LoadRunner (load testing) or FindBugs (java source code analysis for security issues). Manual tests will start by validating functionality based on the requirements. Later stages of system test will include manual end-to-end tests to validate use cases.

6.2 Integration Test Strategy

Because the components will be developed from the bottom-up and top-down, the test strategy will also align to the order of development of components. This will utilize a mostly bottom-up integration test approach, but will also involve the sandwich integration test approach.

Please review the system architecture overview and the subsystem architecture in section 3.

6.3 Test Case alignment with test phases

7. Suspension and resumption
This section specifies the criteria for suspending the testing on the test items associated with the plan. It also specifies the test activities that must be repeated when testing is resumed.

7.1 Automated Unit Test Suite

As components are being developed, unit tests will be developed to test the interfaces of the components and low-level unit tests will be developed to test the methods of the underlying classes in the components.

As a prerequisite to the BAT, the automated unit test suite will be run by the build server on a per-build basis.

When the unit-test suite reports failures, testing will not occur on that build until the failures have been analyzed and resolved. Testing will resume on a build that passes the automated unit test suite.

7.2 Build Acceptance Test (BAT)

When a build is deemed ready to test by development, a build acceptance test will be run on the build. The BAT will consist of a broad but shallow set of tests to determine the overall stability of the build and decide if it is worth testing.

If the BAT fails on a particular build, testing will suspend until another build is created with any BAT failure issues fixed, verified by running the BAT again. Testing will resume on a build that passes the BAT.

Different build acceptance tests will be developed and used for the different test phases. Component BATs will be small and localized for each of the components. Integration BATs will vary based on the level of integration testing being performed. The System Test BAT will contain a set of tests that will utilize each of the components of the system.

7.3 Regression Testing

On a build by build basis, major bug fixes or code changes will be reviewed to determine the effects they may have on the system. If the changes are deemed to cause a sufficient amount of risk, regression test sets of the appropriately judged size will be created and executed.

A system-wide regression will also be run on the release candidate build to ensure incremental changes to the system have not altered the results of the tests that were run early in the test cycle.

7.4 System Design Changes

If at any point in time issues are submitted that require a design change to the system, all testing will be suspended. After the changes to the requirements, system architecture, and object design are made, a review and updates will be performed of the test specifications to ensure they properly align with the revised system changes. After updates are made, testing will resume. Tests in the vicinity of the change must all be rerun. A 20% regression of other tests must also be performed to ensure the changes did not adversely affect other parts of the system.
8. Testing materials (hardware/software requirements)
This section identifies the resources that are needed for testing. This should include the physical characteristics of the facilities, including the hardware, software, special test tools, and other resources needed (office space, etc.) to support the tests.

8.1 Facilities required

The team will need a lab are for the test equipment. The lab area should be approximately 400 sq ft in size, with 160 sq ft of desktop space. The lab area needs multiple power outlets on each side of the room. A table in the center of the room would also allow for easy-access test team technical discussions.

8.2 Hardware required

To enable the team to testing in an optimal environment, the lab needs to contain 4 copies of the system under test. The hardware components in the system [as seen in section 3.3 of the architecture specification], in a single set are a Database Server, a WebServer, a client PC with a web browser that supports Java, and the embedded system that is used in the ambulance. The four systems allow the team to test several components in parallel.

Also, if a single system exhibits a strange bug, it can be left in that state for developer debugging and analysis, while testing continues on the other systems.

8.2 Software required

Software required in this system is minimal. The database server needs the appropriate database (MySQL) installed, setup, and configured properly.

The WebServer machine needs IIS installed the the ISS services started so that it can properly act as a WebServer machine.

The client machine needs Java 1.5.0_b12 installed and properly configured with Internet Explorer 6.0 or newer.

8.3 Special requirements

Additional tools and software may need to be purchased or otherwise acquired or reused. Such software is used to execute special tests that are best run and results recorded and analyzed by automated means. For Load testing this requires a tool like LoadRunner. For Security testing at the complied source code level this requires a tool like FindBugs.
9. Test cases
This section, the core of the test plan, lists the test cases that are used during testing. Each test case is described in detail in a separate Test Case Specification document. Each execution of these tests will be documented in a Test Incident Report document.

9.1 Test specifications derived from nonfunctional requirements

nfrs_3.3.1_usability_MouseKeyboardNavigation
nfrs_3.3.1_usability_EaseOfUnderstandabilityAndUse
nfrs_3.3.1_usability_UserManual
nfrs_3.3.2_reliability_ErrorRecovery
nfrs_3.3.3_performance_DispatchTime3Minutes
nfrs_3.3.3_performance_11MinuteException
nfrs_3.3.3_performance_PageResponseTime15seconds
nfrs_3.3.6_interface_WebBrowserSupport

9.2 Test specifications derived from the use cases in the functional requirements

uc_3.4.2.2_1_Login
uc_3.4.2.2_1_LoginAlternateFlow
uc_3.4.2.2_2_CreateIncident
uc_3.4.2.2_2_CreateIncidentAlternateFlow
uc_3.4.2.2_3_FindIncidentLocation
uc_3.4.2.2_3_FindIncidentLocationAlternateFlow
uc_3.4.2.2_4_LocateAmbulance
uc_3.4.2.2_4_LocateAmbulanceAlternateFlow
uc_3.4.2.2_5_AllocateAmbulance
uc_3.4.2.2_6_AmbulanceTracking
uc_3.4.2.2_7_GetReports
uc_3.4.2.2_8_GeographicAnalysis
uc_3.4.2.2_9_ManageUsers
uc_3.4.2.2_10_DivertToPrivateParty
uc_3.4.5_UINavigation

9.3 Test specifications derived from the system architecture specification

arch_3.3_db_NetworkErr_Login
arch_3.3_db_NetworkErr_CreateIncident
arch_3.3_db_NetworkErr_LocateAmbulance
arch_3.3_db_NetworkErr_Dispatch
arch_3.3_db_NetworkErr_AmbulanceTracking
arch_3.3_db_NetworkErr_GetReports
arch_3.3_db_NetworkErr_ManageUsers

9.4 Test specifications derived from the subsystem decomposition and component diagram

odd_4.3_UserManagement_ValidateUser
odd_4.3_UserManagement_CreateUser
odd_4.3_UserManagement_UpdateUserInfo
odd_4.3_UserManagement_CreateReports
odd_4.4_DispatcherSystem_GetAddress
odd_4.4_DispatcherSystem_GetCoordinates
odd_4.4_DispatcherSystem_GetIncidentDetails
odd_4.4_DispatcherSystem_CreateNewIncident
odd_4.4_DispatcherSystem_LocateAmbulance
odd_4.4_DispatcherSystem_GetNearestHospital
odd_4.4_DispatcherSystem_AllocateAmbulance
odd_4.5_TrackingSystem_TrackAmbulance
odd_4.5_TrackingSystem_CreateMap
odd_4.5_TrackingSystem_SetAmbulanceStatus
odd_4.5_TrackingSystem_UpdateAmbulanceStatus
odd_4.5_TrackingSystem_MonitorTime
odd_4.6_GPSInterface_ProvideRouteDetails
odd_4.6_GPSInterface_ProvideCoordinates
odd_4.6_GPSInterface_CreateMap
odd_4.6_GPSInterface_ProvideStatus
odd_4.7_AddrInterface_FindAddress

9.5 Test case specifications that cover end-to-end test scenarios derived from the use cases

e2e_DispatchAndTrackSingleAmbulanceWhenAvailable
e2e_DispatchAndTrackMultipleAmbulancesWhenAvailable
e2e_DispatchAndTrackSingleAmbulanceWhenBusy
e2e_DispatchAndTrackMultipleAmbulancesWhenBusy
e2e_RequireManualIntervention
e2e_ConcurrentIncidents

10. Testing schedule
This section of the test plan covers responsibilities, staffing and training needs, risks and contingencies, and the test schedule.

10.1 Test Schedule
Test Phase Time Owner
Test Plan Creation 1 wk Test Manager
Test Specification Creation 2 wks Test Leads
Test Specification Team Review 1 wk Project Team
Component Testing 4 wks Component Testers
Integration Testing 4 wks Component & System Testers
System Testing 3 wks System Testers
Performance Testing 1 wk System Testers
Use Case Validation 2 wks System Testers
Alpha Testing 1 wk Product Managers / Analysts
Beta Testing Pilot Program 4 wks Pilot Customers
10.2 Responsibilities

The Test Manager is responsible for the overall test plan (this document) and test resources throughout the course of the project. He needs to be assigned to the project to review the requirements analysis, system architecture design, and object design of the system. From those specifications, he will generate the Test Plan. He will also generate section 9 “Test Cases” in this document. He will generate the list of test specifications and a brief description of each one in this document. He will generate and communicate the test strategy for the project to the test team and the rest of the project team as well as locate, acquire, and/or allocate the proper resources. He will provide periodic updates to the Program Director on the progress of test execution versus the plan as well as the metrics on the quality status of the product, focusing on key issues that need immediate attention from the Project Office.

The Test Leads are responsible for the creation of the detailed test specifications and will generate those and revise section 9 of this document as needed. The leads manage the day-to-day progress of each of their subcomponents and compile and report the metrics to the test manager. They are also responsible for ensuring the testers make adequate progress and follow the overall strategy defined by the Test Manager.

The Component Testers are responsible for the test execution on a daily basis for the component of the system to which they’ve been assigned. They also lead the effort during most of the integration test cycle and hand off the testing to the System Testers during the last states of integration testing.

The System Testers are responsible for functional testing, performance testing, and use case validation testing during the System Test Phase of the project.

During Alpha Testing, the Product Managers / Business Analysts who wrote the requirements will test the system to determine if the system truly aligns with the original vision of the system.

During Beta Testing, a small group of specially chosen customers will pilot the product to ensure it suits their needs.
10.3 Staffing and training needs

1 Test Manager
– responsible for finding and training the following test resources

3 Test Leads
– must be trained on the process being used for this project
– must be trained on the test specification format utilized
– must be trained on the defect/issue tracking system utilized

7 Component Testers
– must know SQL and Java
– must be familiar with NetBeans and Eclipse
– must be skilled at unit testing, API testing, and integration testing
– must be trained on the process being used for this project, the test specification format utilized, and the defect/issue tracking system utilized

7 System Testers
– must know how to use Load testing & Stress testing automation tools (such as LoadRunner)
– must be experienced in system testing and use case validation testing
– must be trained on the process being used for this project.
– must be trained on the test specification format utilized
– must be trained on the defect/issue tracking system utilized
10.4 Risks and contingencies matrix

Risk
Probability
Risk Type
Owner
Contingencies / Mitigation Approach
Unable to acquire the necessary number of skilled personnel as the components become ready to test.
30%
Personnel
Schedule
Test Manager
Resources for components will be split between the existing resources.
Schedule must be adjusted accordingly.
Unable to acquire some of the necessary hardware and software required for integration and system testing
25%
Equipment
Program Manager
Test Manager
Development Manager
Utilize existing acquired hardware.
Split test execution into morning and evening shifts such that testing can occur for multiple teams in the same day using the limited hardware. This requires support of the development during both shifts.
Third party services utilized in the system become unavailable during testing
5%
Third party
Alliance Manager
Setup a communication channel to 3rd party to report and handle issues when they occur.
Use the communication channel above to stay aware of planned outages and maintenance to help schedule test execution.
Components are not delivered on time
25%
Schedule
Development Manager
Integration testing with those components must be delayed until the component is delivered
Overall integration test approach may be modified to do an appropriate amount of bottom-up as well as top-down or sandwich integration.
Schedule must be adjusted accordingly.
Turnover
5%
Personnel
Test Manager
Testers will work in pairs on components. If a single member of the team decides to leave, a secondary testing with the knowledge of the component will still be able to train a new tester or finish the work.
Schedule must be adjusted accordingly.

2007.07.22

Scot Robinson

Final Revisions1. Introduction


This section describes the objectives and extent of the tests. The goal is to provide a framework that can be used by managers and testers to plan and execute the necessary tests in a timely and cost-effective manner.

1.1 Test objectives

The objective of the test suite is to provide adequate coverage metrics, requirements validation, and system quality data such that sufficient data is provided for those making the decision to release.

1.2 Extent of tests

The tests referenced herein are written to validate use cases, requirements (both functional and non-functional), system architecture, and object design. The structured tests for object design will be run first as the components of the system are developed. The structured tests to validate the system architecture will be run next as the system is integrated in bottom-up fashion during integration test.

2. Relationship to other documents

This section explains the relationship of the test plan to the other documents produced during the development effort such as the RAD, SDD, and ODD (Object Design Document). It explains how all the tests are related to the functional and nonfunctional requirements, as well as to the system design stated in the respective documents. If necessary, this section introduces a naming scheme to establish the correspondence between requirements and tests.

2.1 Relationships to documents

Black box tests relating to use cases are developed from the use case diagram(s) in the RAD (requirements analysis document).

Black box tests derived from functional requirements are developed from the requirements lists in the RAD.

Performance tests derived from nonfunctional requirements are developed from the nonfunctional requirements in the RAD.

Structured (unit/white box) tests are generated from the OOD (Object Design Document). The specific tests are developed from the OOD component diagram of each of the components.

Integration tests are developed from the SDD (System/Architecture Design Document). The integration tests generally come from the overall package diagram describing the architecture of the system. The architecture is also used to help in determining the integration test approach.

The test environment (hardware/software) is also derived from the SDD.

A visualization of the relationships.

2.2 Test naming scheme

The names of test cases will indicate from where they have been derived using a system of prefixes. The following prefixes are use to denote the tests were derived from the following places.

uc_ tests derived from use cases

nfrs_ tests derived from nonfunctional requirements

arch_ integration tests derived from the system architecture spec

odd_ structured tests derived from the subsystem decomposition and component diagrams

e2e_ end-to-end systemic tests that exercise the entire user scenarios

If a test was derived from a particular requirement number or component design number, the test names will contain the requirement number or name after prefix followed by an underscore.

After the prefix and the number or name identifier, the test name shall contain a brief, but descriptive name.

3. System overview

This section, focusing on the structural aspects of testing, provides an overview of the system in terms of the components that are tested during the unit test. The granularity of components and their dependencies are defined in this section.

3.1 Software Architecture Overview

* Architecture Overview an excerpt from Deliverable 3: Architecture Specification

3.2 Subsystem Decomposition (Components)

* Subsystem Decomposition an excerpt from Deliverable 3: Architecture Specification

4. Features to be tested/not to be tested

This section, focusing on the functional aspects of testing, identifies all features and combinations of features to be tested. It also describes all those features that are not to be tested and the reasons for not testing them.

4.1 Features to be tested

Components developed in house

The components developed by this organization will be tested unless otherwise noted below in section 4.2.

Components developed by outsource vendors

Components outsourced to be developed specifically for this project where this test team has primary responsibility to test and validate those components will be tested.

Components outsourced to be developed specifically for this project where the outsourced vendor is responsible for development as well as testing will not be tested as components by the component testers. Rather, the test results from the vendor will be reviewed by the test leads, and if they pass review, the component will be tested in-house starting with integration test.

4.2 Items that will not be tested

3rd party and Off-The-Shelf components

It is assumed that 3rd party components were evaluated and the pros and cons properly weighed before choosing that component with our software. The interfaces to those components will be tested, but not the functionality or performance those components.

This includes any 3rd party websites and GPS devices or software.

Infrastructure components

The actual database software utilized is assumed to work as designed and will not be directly tested for functionality. Performance tests will be done during system test with respect to GUI response time that will involve the database. However, no testing will be done directly against the database.

The internet/WIFI backbone will be utilized during testing, however, no tests will be written executed to directly test the communications backbone.

5. Pass/Fail criteria

This section specifies generic pass/fail criteria for the tests covered in this plan. They are supplemented by pass/fail criteria in the test design specification. Note that “fail” in the IEEE standard terminology means “successful test” in our terminology.

5.1 Component Pass/Fail criteria

Tests executed on components only pass when they satisfy the signatures, constraints, and interfaces dictated by the Object Design Specification for that component. This includes positive tests, negative and stress tests, and boundary tests.

If a test exhibits a product failure to meet the objectives of the object design specification, it will fail and a defect/issue will be reported in the defect tracking system for review by the triage team.

5.2 Integration Pass/Fail criteria

Tests executed on integrated components only pass when they satisfy the signatures, constraints, and interfaces dictated by both the object design specification and the system architecture specification. This includes positive tests, negative and stress tests, boundary conditions, and tests that explicitly manipulate the interface environment (such as the physical connection to the database server).

If a test exhibits a product failure to meet the objectives of both the object design specification and the system architecture specification, it will fail and a defect/issue will be reported in the defect tracking system for review by the triage team.

5.3 System Pass/Fail criteria

Tests executed against the system use the functional requirements, non-functional requirements, and use cases as the oracle to determine pass or fail.

If a test exhibits a product failure to meet the objectives of any of the functional requirements, non-functional requirements, or the use cases, it will fail and a defect/issue will be reported in the defect tracking system for review by the triage team.

6. Approach

This section describes the general approach to the testing process. It discusses the reasons for the selected integration testing strategy. Different strategies are often needed to test different parts of the system.

6.1 General Test Strategy

Unit testing and component testing will be performed on the components as they are developed. Test will be executed using test code in the form of either custom test tools or as an automated suite of tests run against the components in their individual sandboxes.

Integrations tests will be performed by both the component testers as well as the system testers. The BAT and the unit test suite will be used as a regression during the integration of components. However, as the integration begins to include GUI level functionality, the tests being run will utilize significantly more manual testing and less automated testing.

System test will require a new set of tools that can measure NFRS compliance, such as LoadRunner (load testing) or FindBugs (java source code analysis for security issues). Manual tests will start by validating functionality based on the requirements. Later stages of system test will include manual end-to-end tests to validate use cases.

6.2 Integration Test Strategy

Because the components will be developed from the bottom-up and top-down, the test strategy will also align to the order of development of components. This will utilize a mostly bottom-up integration test approach, but will also involve the sandwich integration test approach.

Please review the system architecture overview and the subsystem architecture in section 3.

6.3 Test Case alignment with test phases

7. Suspension and resumption

This section specifies the criteria for suspending the testing on the test items associated with the plan. It also specifies the test activities that must be repeated when testing is resumed.

7.1 Automated Unit Test Suite

As components are being developed, unit tests will be developed to test the interfaces of the components and low-level unit tests will be developed to test the methods of the underlying classes in the components.

As a prerequisite to the BAT, the automated unit test suite will be run by the build server on a per-build basis.

When the unit-test suite reports failures, testing will not occur on that build until the failures have been analyzed and resolved. Testing will resume on a build that passes the automated unit test suite.

7.2 Build Acceptance Test (BAT)

When a build is deemed ready to test by development, a build acceptance test will be run on the build. The BAT will consist of a broad but shallow set of tests to determine the overall stability of the build and decide if it is worth testing.

If the BAT fails on a particular build, testing will suspend until another build is created with any BAT failure issues fixed, verified by running the BAT again. Testing will resume on a build that passes the BAT.

Different build acceptance tests will be developed and used for the different test phases. Component BATs will be small and localized for each of the components. Integration BATs will vary based on the level of integration testing being performed. The System Test BAT will contain a set of tests that will utilize each of the components of the system.

7.3 Regression Testing

On a build by build basis, major bug fixes or code changes will be reviewed to determine the effects they may have on the system. If the changes are deemed to cause a sufficient amount of risk, regression test sets of the appropriately judged size will be created and executed.

A system-wide regression will also be run on the release candidate build to ensure incremental changes to the system have not altered the results of the tests that were run early in the test cycle.

7.4 System Design Changes

If at any point in time issues are submitted that require a design change to the system, all testing will be suspended. After the changes to the requirements, system architecture, and object design are made, a review and updates will be performed of the test specifications to ensure they properly align with the revised system changes. After updates are made, testing will resume. Tests in the vicinity of the change must all be rerun. A 20% regression of other tests must also be performed to ensure the changes did not adversely affect other parts of the system.

8. Testing materials (hardware/software requirements)

This section identifies the resources that are needed for testing. This should include the physical characteristics of the facilities, including the hardware, software, special test tools, and other resources needed (office space, etc.) to support the tests.

8.1 Facilities required

The team will need a lab are for the test equipment. The lab area should be approximately 400 sq ft in size, with 160 sq ft of desktop space. The lab area needs multiple power outlets on each side of the room. A table in the center of the room would also allow for easy-access test team technical discussions.

8.2 Hardware required

To enable the team to testing in an optimal environment, the lab needs to contain 4 copies of the system under test. The hardware components in the system [as seen in section 3.3 of the architecture specification], in a single set are a Database Server, a WebServer, a client PC with a web browser that supports Java, and the embedded system that is used in the ambulance. The four systems allow the team to test several components in parallel.

Also, if a single system exhibits a strange bug, it can be left in that state for developer debugging and analysis, while testing continues on the other systems.

8.2 Software required

Software required in this system is minimal. The database server needs the appropriate database (MySQL) installed, setup, and configured properly.

The WebServer machine needs IIS installed the the ISS services started so that it can properly act as a WebServer machine.

The client machine needs Java 1.5.0_b12 installed and properly configured with Internet Explorer 6.0 or newer.

8.3 Special requirements

Additional tools and software may need to be purchased or otherwise acquired or reused. Such software is used to execute special tests that are best run and results recorded and analyzed by automated means. For Load testing this requires a tool like LoadRunner. For Security testing at the complied source code level this requires a tool like FindBugs.

9. Test cases

This section, the core of the test plan, lists the test cases that are used during testing. Each test case is described in detail in a separate Test Case Specification document. Each execution of these tests will be documented in a Test Incident Report document.

9.1 Test specifications derived from nonfunctional requirements

nfrs_3.3.1_usability_MouseKeyboardNavigation

nfrs_3.3.1_usability_EaseOfUnderstandabilityAndUse

nfrs_3.3.1_usability_UserManual

nfrs_3.3.2_reliability_ErrorRecovery

nfrs_3.3.3_performance_DispatchTime3Minutes

nfrs_3.3.3_performance_11MinuteException

nfrs_3.3.3_performance_PageResponseTime15seconds

nfrs_3.3.6_interface_WebBrowserSupport

9.2 Test specifications derived from the use cases in the functional requirements

uc_3.4.2.2_1_Login

uc_3.4.2.2_1_LoginAlternateFlow

uc_3.4.2.2_2_CreateIncident

uc_3.4.2.2_2_CreateIncidentAlternateFlow

uc_3.4.2.2_3_FindIncidentLocation

uc_3.4.2.2_3_FindIncidentLocationAlternateFlow

uc_3.4.2.2_4_LocateAmbulance

uc_3.4.2.2_4_LocateAmbulanceAlternateFlow

uc_3.4.2.2_5_AllocateAmbulance

uc_3.4.2.2_6_AmbulanceTracking

uc_3.4.2.2_7_GetReports

uc_3.4.2.2_8_GeographicAnalysis

uc_3.4.2.2_9_ManageUsers

uc_3.4.2.2_10_DivertToPrivateParty

uc_3.4.5_UINavigation

9.3 Test specifications derived from the system architecture specification

arch_3.3_db_NetworkErr_Login

arch_3.3_db_NetworkErr_CreateIncident

arch_3.3_db_NetworkErr_LocateAmbulance

arch_3.3_db_NetworkErr_Dispatch

arch_3.3_db_NetworkErr_AmbulanceTracking

arch_3.3_db_NetworkErr_GetReports

arch_3.3_db_NetworkErr_ManageUsers

9.4 Test specifications derived from the subsystem decomposition and component diagram

odd_4.3_UserManagement_ValidateUser

odd_4.3_UserManagement_CreateUser

odd_4.3_UserManagement_UpdateUserInfo

odd_4.3_UserManagement_CreateReports

odd_4.4_DispatcherSystem_GetAddress

odd_4.4_DispatcherSystem_GetCoordinates

odd_4.4_DispatcherSystem_GetIncidentDetails

odd_4.4_DispatcherSystem_CreateNewIncident

odd_4.4_DispatcherSystem_LocateAmbulance

odd_4.4_DispatcherSystem_GetNearestHospital

odd_4.4_DispatcherSystem_AllocateAmbulance

odd_4.5_TrackingSystem_TrackAmbulance

odd_4.5_TrackingSystem_CreateMap

odd_4.5_TrackingSystem_SetAmbulanceStatus

odd_4.5_TrackingSystem_UpdateAmbulanceStatus

odd_4.5_TrackingSystem_MonitorTime

odd_4.6_GPSInterface_ProvideRouteDetails

odd_4.6_GPSInterface_ProvideCoordinates

odd_4.6_GPSInterface_CreateMap

odd_4.6_GPSInterface_ProvideStatus

odd_4.7_AddrInterface_FindAddress

9.5 Test case specifications that cover end-to-end test scenarios derived from the use cases

e2e_DispatchAndTrackSingleAmbulanceWhenAvailable

e2e_DispatchAndTrackMultipleAmbulancesWhenAvailable

e2e_DispatchAndTrackSingleAmbulanceWhenBusy

e2e_DispatchAndTrackMultipleAmbulancesWhenBusy

e2e_RequireManualIntervention

e2e_ConcurrentIncidents

10. Testing schedule

This section of the test plan covers responsibilities, staffing and training needs, risks and contingencies, and the test schedule.

10.1 Test Schedule

Test Phase Time Owner

Test Plan Creation 1 wk Test Manager

Test Specification Creation 2 wks Test Leads

Test Specification Team Review 1 wk Project Team

Component Testing 4 wks Component Testers

Integration Testing 4 wks Component & System Testers

System Testing 3 wks System Testers

Performance Testing 1 wk System Testers

Use Case Validation 2 wks System Testers

Alpha Testing 1 wk Product Managers / Analysts

Beta Testing Pilot Program 4 wks Pilot Customers

10.2 Responsibilities

The Test Manager is responsible for the overall test plan (this document) and test resources throughout the course of the project. He needs to be assigned to the project to review the requirements analysis, system architecture design, and object design of the system. From those specifications, he will generate the Test Plan. He will also generate section 9 “Test Cases” in this document. He will generate the list of test specifications and a brief description of each one in this document. He will generate and communicate the test strategy for the project to the test team and the rest of the project team as well as locate, acquire, and/or allocate the proper resources. He will provide periodic updates to the Program Director on the progress of test execution versus the plan as well as the metrics on the quality status of the product, focusing on key issues that need immediate attention from the Project Office.

The Test Leads are responsible for the creation of the detailed test specifications and will generate those and revise section 9 of this document as needed. The leads manage the day-to-day progress of each of their subcomponents and compile and report the metrics to the test manager. They are also responsible for ensuring the testers make adequate progress and follow the overall strategy defined by the Test Manager.

The Component Testers are responsible for the test execution on a daily basis for the component of the system to which they’ve been assigned.

They also lead the effort during most of the integration test cycle and hand off the testing to the System Testers during the last states of integration testing.

The System Testers are responsible for functional testing, performance testing, and use case validation testing during the System Test Phase of the project.

During Alpha Testing, the Product Managers / Business Analysts who wrote the requirements will test the system to determine if the system truly aligns with the original vision of the system.

During Beta Testing, a small group of specially chosen customers will pilot the product to ensure it suits their needs.

10.3 Staffing and training needs

1 Test Manager

– responsible for finding and training the following test resources

3 Test Leads

– must be trained on the process being used for this project

– must be trained on the test specification format utilized

– must be trained on the defect/issue tracking system utilized

7 Component Testers

– must know SQL and Java

– must be familiar with NetBeans and Eclipse

– must be skilled at unit testing, API testing, and integration testing

– must be trained on the process being used for this project, the test specification format utilized, and the defect/issue tracking system utilized

7 System Testers

– must know how to use Load testing & Stress testing automation tools (such as LoadRunner)

– must be experienced in system testing and use case validation testing

– must be trained on the process being used for this project.

– must be trained on the test specification format utilized

– must be trained on the defect/issue tracking system utilized

10.4 Risks and contingencies matrix

Risk

Probability

Risk Type

Owner

Contingencies / Mitigation Approach

Unable to acquire the necessary number of skilled personnel as the components become ready to test.

30%

Personnel

Schedule

Test Manager

Resources for components will be split between the existing resources.

Schedule must be adjusted accordingly.

Unable to acquire some of the necessary hardware and software required for integration and system testing

25%

Equipment

Program Manager

Test Manager

Development Manager

Utilize existing acquired hardware.

Split test execution into morning and evening shifts such that testing can occur for multiple teams in the same day using the limited hardware. This requires support of the development during both shifts.

Third party services utilized in the system become unavailable during testing

5%

Third party

Alliance Manager

Setup a communication channel to 3rd party to report and handle issues when they occur.

Use the communication channel above to stay aware of planned outages and maintenance to help schedule test execution.

Components are not delivered on time

25%

Schedule

Development Manager

Integration testing with those components must be delayed until the component is delivered

Overall integration test approach may be modified to do an appropriate amount of bottom-up as well as top-down or sandwich integration.

Schedule must be adjusted accordingly.

Turnover

5%

Personnel

Test Manager

Testers will work in pairs on components. If a single member of the team decides to leave, a secondary testing with the knowledge of the component will still be able to train a new tester or finish the work.

Schedule must be adjusted accordingly.

Advertisements
Skip to toolbar