Building Effective Measurement Solutions Workshop

Measurement Challenges

• Goals for measurement
• Tell a story of the value of testing
• Estimate number of test cases for test planning
• Enterprise level measurement
• Standards across multiple teams

Reasons for Measurement

• Drive improvement 
• Planning of testing
• Demonstrate value of testing
• Provide testing status and control
• Improve testing efficiency

How to Establish Enterprise Metrics

• Different areas have different goals
• Different areas have different challenges
• Overarching goal - make more money
• Investment in automation
• Reduction in cost (through offshoring)
• Reduction in time to market
• Create a pallet of metrics

Group 1: Demonstrating the Value of Testing

Name of Metric:             Test ROI

Description:                   Value of doing testing

Formula:                       (Benefit - Cost) / Cost

Measures:                     Test Efficiency
                                        Time to Test
                                        Time to Fix

Source:                         Defect counts
                                       Time to Test = (Test Time) x (% Test Complete)
                                       Test Efficiency = (# of defects in test) / (total defects)

                                       Benefit=Quality product
                                       Business efficiency
                                       Meet requirements
                                       Legislative compliance

                                      Costs= Cost to test
                                      Cost to fix
                                      Cost to re-test             

 Back to top

Group 2: Providing Testing Status and Control

Name of Metric:             % test cases executed

Description:                   Test execution progress (point in time)

Formula:                       (# test cases run) / (total # test cases planned)

Measures:                     # test cases
                                       # test cases executed
                                       Time to Fix

Source:                          Test Management Tool

When gathered:            During execution cycle

When reported:             Monitored daily
                                         Reported weekly

Analysis:                       Redirect resources

Dashboard:                 % test cases executed - where we are at now
                                       1st run fail rate - how much more testing will be needed
                                       % test cases blocked - how much more to do

Back to top

Group 3: Improve Testing Efficiency

Name of Metric:             Improvement (Efficiency)

Description:                  Established Baseline - Monitored change => improvement or not
                                        How do we do earlier defect discovery in smoke tests?

Formula:                       (defects) x (effort) / size = defect hours per unit of work

Measures:                     # of defects in 1st run
                                      When defect discovered (which phase)
                                      Size (e.g. dev. hrs. for change, lines of code, function points)
                                      Effort (# of testers)                              

Source:                        Defects <= Quality Center
                                      Phase   <= Quality Center
                                      Size      <= # of requirements
                                      Effort   <= E-Time (time reporting system)

Analysis:                      

1. Establish Baseline
1. Frequency of collection - every time we run smoke test or on demand
2. Frequency of reporting - monthly
3. Who / how to collect?
4. Tools to use / record
5. Collect 3-4 months and establish baseline (mean)

1. Measure / Review Improvement
1. Threshold? + or - 3 standard deviations
2. ^ threshold = go look at change

Reference:        W. Ed. Deming "Out of the Crisis"

Back to top

Post-Conference: Agile Panel Discussion | Better Test Planning Solutions Workshop | Risk Management Workshop
Building Effective Measurement Solutions Workshop | Challenges of Test Automation Solutions Workshop