Home | About QAI | QAI Canada Home Page | Federated Chapters
Conference Home Page
Manager's Workshop
QUEST Magazine
Get Involved



Challenges of Test Automation Solutions Workshop


Topic - Tool Selection and Evaluation
           

How to

  • See objects of application
  • Research & evaluation  -  requirements
  • Cost - Consider the price relative to the realistic return-on-investment which can be realized.
  • Try it - Get detailed demonstrations and conduct a proof-of-concept.
  • Go through the pain - There are always challenges.  Try to identify the pain to expect before you purchase.
  • Types of automation - Functional testing, performance testing, application security testing, other?
  • Resources (time)
Constraints
  • Time, vendor, business
  • Cost - Budget often dictates tool to be selected (as opposed to capabilities of tool).
  • Resources - Will additional resource be needed to implement the tool?  If so, are they available?
  • Vendor knowledge & availability - What's the supply of contract resources and knowledge available?  If there's a short supply will the current team be able to learn the tool so that it's used and doesn't become "shelfware".
  • Team skill set - Must take into account existing skill set of existing resources and their ability to learn new tool.
  • Availability of applications to automate - How does the tool handle the dynamics of an application especially if the application is still in a state of evolution?
Assumptions
  • With proper due diligence, the solution will be determined - A POC (proof-of-concept) is essential.
  • Support - The tool(s) being considered has support and maintenance offered and available by vendor.
  • Resources
  • See constraints
Open Source vs. COTS
  • Cost - Open source is typically assumed to be less expensive.  They may or may not exceed the R.O.I. potential of COTS solutions.
  • Customization - It's typically considered that any automation solution would require some level of customization and/or configuration.
  • Technical capability
  • Support - COTS solutions typically offer a greater degree of vendor support.
  • Political situation / environment - Does management already have a tool in mind?  Is the decision already "made"?  How much convincing and persuasion is required to sell management on the need for an automation tool?
  • Culture
End Results
  • Works as required - The goal is to obtain an automation solution which meets expectations.  It may not meet all needs, but it meets most of them.
  • Clear data on choice
  • Investment requirement - To be successful with any tool requires time, effort, and money.  The money spent should be thought of as an investment.
  • Timing, cost, resources
  • Vendor relationship - Ensure that a relationship is developed at the beginning of the process and continues after the tool has been selected.
  • What can and cannot be automated - This is a key factor in determining level of success.  Know what the tool can and cannot automate and ensure that it meets your needs.

Back to top

Topic - How to Find and Retain Automation Resources           

How to

  • Offshore outsourcing - Always an option.  Typical direct costs (bill rates) may be lower but other challenges may present themselves thus driving the overall costs higher - project takes longer (requiring more hours), more involvement from management and other technical experts may be needed, breakdown in communication, etc.
  • Internal references - Use existing base of employees and resources to find other resources with automation expertise.  Referrals are always best.
  • Student recruitment - Look at colleges, job fairs, conferences, as a recruiting source.
  • Incentives (salary, bonus) - Automation expertise adds value to resources.  They should be compensated adequately to better retain resources.
  • Type of automation - During the search for automation experts ensure that you know the type of testing expertise required.  Different types of software testing typically require different experience and knowledge.  Are we looking for software performance experience and knowledge?  Functional testing?  Application security testing?
  • Resources (time)
Constraints
  • Validation of deliverables - If utilizing offshore or outsourcing of any type ensure that deliverables are well-defined.
  • Short term employment (students, consultants)
  • Team integration of subject matter experts
  • Versioning of changes to automation
Assumptions
  • Specialized skills - While test automation requires (manual) testing expertise and experience it also requires a different type and level of skill.  It most likely requires design and development (programming) knowledge.
  • Short term employment (students, consultants)
End Results
  • Combination of skills
    - scripting
    - building of repository of test cases
  • Reduce overhead cost
  • Maximum coverage of testing applications - It's a goal that, utilizing test automation, more conditions and scenarios be tested.  Meeting this objective provides greater coverage of test requirements.

Back to top

Topic - How to Automate           

Deciding Factors

  • Understand the goal of the test - What is being verified and validated?  Is the focus on performance, and/or functional, and/or application security?
  • Cost VS benefits - Measure and track the return-on-investment.  If an ROI is not realized it doesn't matter what the solution cost.
  • Where are the spending efforts now?
  • How complicated is the project? - Knowing this will help estimate the "learning curve".  Complex projects are not the best "kick-off point" for the implementation of automation tools.  Start simple.
  • Time Constraints - Are there critical target dates that must be met?  If so, and test automation has not been implemented, it may not be the right time to start with test automation and begin an automation framework.
  • Increase test coverage - Has poor test coverage (of requirements) been an issue in the past?
Challenge:
  • User but-in
  • What to automate - To often organizations purchase an automation tool and adopt the attitude that an "entire" test project should be automated.  This is seldom, if ever, realistic.  Know what to automate and what to leave manual.  Time can be wasted trying to automate test cases which will only be executed once or twice more.
  • Approach / Framework - Have a strategy.
  • Test data Control
Constraints
  • Skills & resources available - Obtain at least a few resources that have skill and experience in test automation.  Skill requires not only technical expertise but also an understanding of different methodologies to utilize when implementing an automation tool.
  • Time available - Critical target dates will play an important factor in deciding which strategies to use when implementing automation.  There are various approaches that can be taken.
  • Configurations/Environments
Assumptions:
  • Management buy-in / available staff - It is assumed that if an organization has purchased and decided to implement an automation solution then that organization's management and team are committed to its success.  If not, failure is almost certain.
  • Learned skill / processes will improve - It takes time.  Over time the team will improve its' skill and the framework will evolve.
End Result:
  • Large set of automation candidates - Automate as much as possible in regards to that which can be and should be automated.  Do not get carried away and try to automate everything.
  • Sets you know you will not automate

Back to top


Post-Conference: Agile Panel Discussion | Better Test Planning Solutions Workshop | Risk Management Workshop
Building Effective Measurement Solutions Workshop
| Challenges of Test Automation Solutions Workshop


Quality Engineered Software & Testing (QUEST) Conference - Copyright © 2008
www.qaiquest.org