Home | About QAI | QAI Canada Home Page | Federated Chapters
Conference Home Page
Manager's Workshop
QUEST Magazine
Get Involved

Building Effective Measurement Solutions Workshop

Measurement Challenges

  • Goals for measurement
  • Tell a story of the value of testing
  • Estimate number of test cases for test planning
  • Enterprise level measurement
  • Standards across multiple teams

Reasons for Measurement

  • Drive improvement 
  • Planning of testing
  • Demonstrate value of testing
  • Provide testing status and control
  • Improve testing efficiency

How to Establish Enterprise Metrics

  • Different areas have different goals
  • Different areas have different challenges
  • Overarching goal - make more money
  • Investment in automation
  • Reduction in cost (through offshoring)
  • Reduction in time to market
  • Create a pallet of metrics

Group 1: Demonstrating the Value of Testing

Name of Metric:             Test ROI

Description:                   Value of doing testing

Formula:                       (Benefit - Cost) / Cost

Measures:                     Test Efficiency
                                        Time to Test
                                        Time to Fix

Source:                         Defect counts
                                       Time to Test = (Test Time) x (% Test Complete)
                                       Test Efficiency = (# of defects in test) / (total defects)

                                       Benefit=Quality product
                                       Business efficiency
                                       Meet requirements
                                       Legislative compliance

                                      Costs= Cost to test
                                      Cost to fix
                                      Cost to re-test             

 Back to top

Group 2: Providing Testing Status and Control

Name of Metric:             % test cases executed

Description:                   Test execution progress (point in time)

Formula:                       (# test cases run) / (total # test cases planned)

Measures:                     # test cases
                                       # test cases executed
                                       Time to Fix

Source:                          Test Management Tool

When gathered:            During execution cycle

When reported:             Monitored daily
                                         Reported weekly

Analysis:                       Redirect resources

Dashboard:                 % test cases executed - where we are at now
                                       1st run fail rate - how much more testing will be needed
                                       % test cases blocked - how much more to do

Back to top

Group 3: Improve Testing Efficiency

Name of Metric:             Improvement (Efficiency)

Description:                  Established Baseline - Monitored change => improvement or not
                                        How do we do earlier defect discovery in smoke tests?

Formula:                       (defects) x (effort) / size = defect hours per unit of work

Measures:                     # of defects in 1st run
                                      When defect discovered (which phase)
                                      Size (e.g. dev. hrs. for change, lines of code, function points)
                                      Effort (# of testers)                              

Source:                        Defects <= Quality Center
                                      Phase   <= Quality Center
                                      Size      <= # of requirements
                                      Effort   <= E-Time (time reporting system)


  1. Establish Baseline
    1. Frequency of collection - every time we run smoke test or on demand
    2. Frequency of reporting - monthly
    3. Who / how to collect?
    4. Tools to use / record
    5. Collect 3-4 months and establish baseline (mean)
  1. Measure / Review Improvement
    1. Threshold? + or - 3 standard deviations
    2. ^ threshold = go look at change

Reference:        W. Ed. Deming "Out of the Crisis"

Back to top

Post-Conference: Agile Panel Discussion | Better Test Planning Solutions Workshop | Risk Management Workshop
Building Effective Measurement Solutions Workshop
| Challenges of Test Automation Solutions Workshop


Quality Engineered Software & Testing (QUEST) Conference - Copyright © 2008