Conference Paper: In-the-Wild Testing: The Case for Real-World QA

Authored by Doron Reuveni, CEO and co-founder of uTest

For as long as there’s been software, there has been software testing. As the industry matured, it segmented testing into various schools and methods; manual vs. automated, in-house vs. outsourced; guided vs. exploratory; emulators vs. remote access. In each case, these innovations take place inside the confines of the QA lab, either within the company firewall, or in an office halfway around the world. Starting to see the problem?

When companies wanted to improve their testing, they did so within this somewhat sterile environment. And yet, even with millions spent on QA in all of its various forms, organizations continue to launch products that miss deadlines, exceed budgets and – most importantly – don’t work as designed in the hands of their actual users.

Our thesis: the problem has little to do with in-the-lab testing practices, methodologies or budget. Rather, there is a fundamental link missing in the QA chain: in-the-wild-testing. After all, users consume web, desktop and mobile applications under:

  • Adverse, unpredictable and widely varied environments
  • Outdated browsers, plug-ins and third party apps
  • Unique hardware and devices
  • Imperfect connectivity (both Wi-Fi and mobile carriers)

The only way to launch apps that consistently work in the hands of users – apps that are functional, reliable, secure and intuitive – is to move a portion of testing out of the lab and into the wild. This means involving professional testers, with real devices, operating under true real-world conditions. In this way, organizations can make sure that all of that great testing they did translates into a superior end user experience.

So what makes in-the-wild testing so distinct from the testing methods discussed earlier? Here are a few key differentiators:

  • Mirror Real-World Conditions: While this attribute pertains to all testing types, it is perhaps most applicable to usability and localization testing. For instance, suppose your target users are mothers, ages 35-45, who live in Latin American. By moving your testing into the wild – with a handpicked group of testers that match your exact demographics– you get a much clearer picture of how your target users will respond to your application. Essentially, it’s like running a beta test, except one that is extremely specific and only includes professional testers.
  • Identify Fringe Use-Cases: When testing a web application, for instance, it’s fairly common to have your QA team verify its functionality across all of the major browsers. But what about the various third-party applications (e.g. anti-virus, plug-ins, etc.) that mostly exist on the hardware of your users, and not your QA team. With in-the-wild testing, you get insight into the unusual use cases that can lead to big problems post launch.
  • Test On-Demand: Unlike most QA projects, in-the-wild testing is designed to be utilized where and when you need it most, requiring very little setup time. This benefits companies whose QA requirements change frequently (usually those adhering to an agile framework).

Key Benefits of Testing “In the-Wild”

In today’s world of pay-as-you-go products, any software bugs that make it in front of your users will immediately decrease usage, dragging revenue down with it. Besides the financial incentives, in-the-wild testing offers other key benefits, including:

  • Maintain Control: One of the greatest concerns with testing in the wild is a perceived loss of control, and thus, quality. But done correctly, the in-house QA leader can maintain complete control and oversight of the entire test cycle.
  • Increase App Quality: By testing in the wild, a development team can receive a list of bugs 3-4 weeks before they would have normally surfaced – giving them more time to launch a finished product.
  • Tester Diversity: Testing in-the-wild gives you the opportunity to off-set the group-think that often plagues so many internal QA teams. This is particularly helpful in terms of usability, where you can involve testers who are totally unfamiliar with your product.
  • Improve Efficiency: Since it is an on-demand solution, crowdsourcing helps alleviate the pains associated with peak release times by leveraging a community of testers exactly when you need them at any point in the SDLC.

While new to some, in-the-wild testing is an established practice inside many of the world’s most successful companies. Here are a few of its more vocal proponents:

  • Shie Erlich, Testing Manager, Microsoft: “You wouldn’t believe some of the behaviors we observed on these home machines. When you are testing for performance, it’s imperative to know how the software runs outside of the lab environment.”
  • Michael Cooper, Director of QA, T-Mobile: “Test on the device itself, as soon as possible; you can miss a lot of defects if you only test on an emulator.”
  • James Whittaker, Director of Test Engineering, Google: “Automation is good at analyzing data and noticing patterns. It is not good at determining relevance and making judgment calls. Fortunately humans excel at judgment.”

The trend is clear: Companies of all shapes and sizes are leveraging in-the-wild testing to ensure a higher level of quality. Those who ignore this real-world component do so at their own risk. Choose wisely!

Discuss this conference paper online as a member of the QUEST LinkedIn Group.