Every software development company focuses on developing quality software. The only way to track the software quality is
evaluating it at every stage of its development. It requires some kind of metrics, which is obtained through effective
testing methods. Each stage of software testing is effectively monitored for the software QA.
1. Software measurements are used for:
2. Deriving basis for estimates
3. Tracking project progress
4. Determining (relative) complexity
5. Understanding the stage of desired quality
6. Analyzing defects
7. Validating best practices experimentally
Here, some software testing metrics are proposed for black box testing that has real world applications. It discusses:
Importance of software testing measurement
Different techniques/processes for measuring software testing
Metrics for analyzing testing
Methods for measuring/computing the metrics
Advantages of implementing these metrics
These metrics helps in understanding the inadequacies in different software QA stages and finding better correcting
What is measurement and why it is required?
The process of assigning numbers or symbols to attributes of real world entities for describing them according to defined
rules is called measurement.
For developing quality software, several characteristics like requirements, time and effort, infrastructural cost,
requirement testability, system faults, and improvements for more productive resources should be measured.
Measuring software testing is required:
1. If the available test cases cover all the systems aspects
2. For tracking problems
3. For quantifying testing
Choose the suitable metrics
Several metrics can measure software-testing process.
Here, the following types of metrics are identified:
These raw data are collected in a testing effort and applied in formulae used to derive Calculated Metrics.
The Test Metrics comprise of the Number of
Test Cases Passed, Failed, Under Investigation, Blocked, Re-executed and Test Execution Time.
They convert the Base Metrics data into useful information. Every test efforts must implement the following Calculated
% Defects Corrected
% Test Coverage
% Test Cases Passed & Blocked
% Test Effectiveness & Efficiency
% 1st Run Failures
Defect Discovery Rate
Defect Removal Cost
Measurements for Software Testing
The corresponding software testing process in software development measures each step for ensuring quality product delivery.
1. Software Size:
The amount of functionality of an application determines this and is calculated by
Function Point Analysis
Task Complexity Estimation Methodology
2. Requirements review:
Before software development, the Software requirement specifications (SRS) from the client are obtained. It must be:
The Review Efficiency is a metric that offers insight on the review quality and testing.
Review efficiency=100*Total number of defects found by reviews/Total number of project defects
3. Effectiveness of testing requirements:
It is measured by maintaining Requirement Trace-ability matrix and specification of requirements, which should have:
SRS Objective, purpose
Data Structures/Elements Safety
Constraints & limitations
Next comes the updating of the crucial requirement trace-ability matrix or RTM, which determines the number and types of
While measuring the mapping of test cases, the number and priority of requirement it tests, its execution effort and
requirement coverage must be determined.
The Requirement compliance factor (RCF) measures the coverage provided by the test cases to one or set of requirement(s).
j is a set of requirements and (j=1-m);
Xi=2, if the test case (say Tj) tests requirements Ri completely,
=1, if it tests partially,
=0, if otherwise.
Effectiveness=RCFj/Ej where Ej=Time required for executing a test case
4. Evaluating estimation accuracy
Relative error=(A-E)/A where E is estimate of a value and A is actual value.
For a collection of estimates, the mean RE for n projects is
For a set of n projects, the mean magnitude of RE (MRE) is
Of a set of n projects, an acceptable level for MRE is less than 0.25.
If K is the number of projects whose mean magnitude of relative error is less than or equal to q,
then the prediction quality pred(q)=K/n
5. Measurement of Efficiency in testing process
In software testing, we must keep tabs on what we had planned and what we have actually achieved for measuring efficiency.
Here, the following attributes play major roles: -
Cost: The Cost Variance (CV) factor measures the risk associated with cost.
CV=100*(AC - PC)/PC, AC=Actual Cost, PC=Planned/Budgeted Cost.
Effort: Effort Variance (EV) measures effort.
EV=100*(AE - PE)/PE
(AE=Actual Effort, PE=Planned Effort)
Schedule: Schedule Variance (SV) is important for project scheduling.
SV=100*(AD-PD)/PD where AD=Actual duration and PD=Planned duration.
Cost of quality: It indicates the total effort expended on prevention, appraisal and rework/failure activities versus all
Prevention Effort=Effort expended on planning, training and defect prevention.
Appraisal Effort=Effort expended on quality control activities.
Failure effort=Effort expended on rework, idle time etc.
COQ=100*(PE + AE + FE)/Total project effort.
Size variance: It is the degree of variation between estimated and actual sizes.
Size Variance=100*(Actual Software Size-Initial Estimated Software Size)/Initial Estimated Software Size
Defect density: It is the total number of defects in software with respect to its size.
Defect density=Total number of defects detected/software size
Mean Time Between Failures: MTBF is the mean time between two critical system failures or breakdowns.
MTBF=Total time of software system operation/Number of critical software system failures.
Defects: Defects are measured through:
Defect distribution: It indicates the distribution of total project defects.
Defect Distribution=100*Total number of defects attributed to the specific phase/Total number of defects.
Defect removal effectiveness: Adding the number of defects removed during the phase to the number of defects found later
Benefits of implementing metrics in software testing:
Improves project planning.
Understanding the desired quality achieved.
Helps in improving the processes followed.
Analyzing the associated risks.
Improving defect removal efficiency.