Home | About QAI | QAI Home Page | Federated Chapters
QuEST Chicago

Conference Home Page

QUEST Magazine



Solutions Workshop Notes for
"Building Effective Measurement Solutions"

Date:  4/30/08
Facilitator:  Nancy Kastl
Co-Facilitators:  Barbara Ainsworth and Sharon Reinhard
Track Host:  Luke Musinski


1.  What are the best testing metrics for management?

Constraints

  • Budget/Resources
  • Different Data Collection Tools
  • Time Constraints

Assumptions

  • Data is available
  • Audience is upper management
  • System Testing completed

Success criteria

  • Metrics needs to be reported to management in terms of dollars.  Therefore, identifying the Cost of a Defect up front (i.e. $1 if found in requirements, $10 if found in testing, $100 if found in production) would help to tie the metrics such as where found to a dollar amount that management could relate to.                                

Solution Approaches

  • % Complete (Test Case)
  • % Complete (over time line)
  • Defect Density (per module)
  • Find/Fix (Trending)
  • Root Cause (where found/when)
  • Pass/Fail Rate over Time
  • Aging Report
  • Avg. time to fix
  • Code Coverage
  • Reactivation (Retests fail)
  • Schedule metrics
  • Cost metrics (cost of a defect - more expensive if found in system testing or production)

Back to Top

2.  What is the best way to get the data for measurement?

Constraints

  • Getting accurate data
  • Real-Time data
  • Time-Frame for data collection
  • Budget constraints

Assumptions

  • Data is being collected
  • There is a need to collect data
  • We know the right thing to collect
  • We know the right thing to collect

Success Factors

  • Timeframes are reasonable and achievable
  • Budget to allow the use of automated tools

Solution Approaches

  • Training on Std. data collection process
  • Training on Std. processes
  • Centralized Repository for data
  • Having the right tools
  • Automate where possible

Back to Top

3.  What are the quality metrics?

Constraints

  • Depends on product, project, service, or process to be measured
  • Need a common definition of quality
  • Getting accurate data
  • Having a sizing measure for software (LOC or Function Points)

Assumptions

  • Want to measure the quality of production applications
  • Will be able to define the quality characteristics that are important for the production application
    - Lack of bugs
    - Customer satisfaction
    - Meets requirements
    - User friendly
    - On time
    - Within budget
    - Software performance
    - Maintainability
    - Etc.

Success Factors

  • Agreement to the definition of quality
  • Agreement to relative importance of quality characteristics

Solution Approaches

  • Identify if you are measuring the quality of a product, project, process, or service
  • Define the characteristics/dimensions of quality for the item
  • Rank the relative importance of each quality characteristic/dimension
    (e.g. is software performance more important than user friendly?)
  • Determine the calculation or rating for each quality characteristic
    (e.g. customer satisfaction is a rating; user friendly is a score; meets requirements is a percentage)
  • Determine if the data is available
  • Determine the reporting (who, what, when, how)
  • Pilot the metrics and refine based on experience

Back to Top

4. How do you create a simple and meaningful balanced scorecard/dashboard?

Constraints

  • Data integrity
  • Data source automated
  • Management support
  • Time and funding to develop all requirements for the balanced scorecard/dashboard

Assumptions

  • Publish to a wide audience
  • Not a Project Life Cycle - the focus is on testing/QA only
  • Use prior and current data
  • Publish the scorecard/dashboard weekly

Solution Approaches

            Quality Perspective:

  • Number of defects per project
  • Number of defects per phase
  • Coverage per project
  • Defect causes

            Financial Perspective:

  • Number of defects post implementation
  • QA budget
  • SLA penalities

            People Perspective:

  • Training
  • Certifications

            Customer Perspective:

  • SLAs met
  • Interview results
  • Post mortems feedback

Back to Top

5. How do you show that QA adds value and reliability?

Constraints

  • Culture of finger pointing and blame
  • No incentives for upper management to fund tool(s) or go through the QA Process
  • Most times cannot prove value and reliability through testing
  • No consistency on what to measure (I.e. project standards differ between small and large projects; can be treated differently but can be categorized)

Assumptions

            Culture where senior management:

  • Jumps to conclusions
  • Does not understand
  • Does not trust data or test methods
  • Does not understand how metrics will help them

Solution Approaches

  • Educate senior managers and obtain buy-in
  • Establish standards
  • Tools for report/data
  • Gain consensus on defining what to measure and what to do when/with the data
  • Learn and improve the process with data
  • Vendor/outsource for best practice

Back to Top

6. Bar Stool Problem: What are some alternate avenues, other then escalation to management, to ensure service level agreements (SLAs), in regards to fixing defects, are met by applicable parties?

Solutions

  • Ensure that the defined SLAs are reasonable by opening the lines of communication amongst applicable parties.  This can help with understanding expectations and defining a common ground to stand on.
    • For example, is it feasible to expect severity 1 defects to be assigned a root cause in 24 hours and fixed in 48 hours?  If not, then redefine SLAs with input from all applicable parties.
  • Ensure that defects are assigned to a particular person and not just sent to a queue.  This may help avoid defects being overlooked in the queue.
  • If possible, assign a dedicated person from the applicable party whose sole responsibility is to address and fix defects.
  • Promote a positive environment by rewarding resources that meet SLAs.  This may encourage others to live into SLAs.
  • Produce a daily report that is posted near the project team so everyone can visually account for project defect status.  For example, use a chart that is color coded with green, yellow, and red to help identify defects statuses.  This will help communicate without having to say anything.
  • Have daily defect status reports meetings with applicable parties to address what was done yesterday, what is being done today, and what is slated for tomorrow.

Back to Top

Workshops & Panel: Test Management | Measurement | Solutions Workshop
Agile Panel Discussion [Cox] [Rinko-Gay]


Quality Engineered Software & Testing (QUEST) Conference - Copyright © 2008
www.qaiquest.org