Towards the end of 2020, many enterprises started pivoting towards Test less trend. However, not all teams within an enterprise were transforming at the same speed. Several teams continue testing the entire regression suite without being sure if they should really test all of it for a requirement. Regression test suites have evolved over the years and, in some cases, more than a decade. Because of the increasing number of features and constant changes in requirements, organizations continue to add test cases to the existing test suite. Testers who wrote these test cases have moved on, and very few people know how much is relevant.

There is hardly any proactive action to remove obsolete test cases because of the fear of losing critical test cases. Common problems faced by clients are:

•  “What if something changes midway during the sprint?”

•  “What if I discover a bug on the last day of sprint?”

•  “What if edge cases and historical learnings are overlooked due to current priorities?”

Analyzing historical defects can help identify the right set of test cases that must give sufficient coverage while reducing defect leakage. Research suggests that the next wave in the QA market is driving towards intelligent automation and AI and ML. Beyond test automation, unless most of the testing life cycle phases are free of human intervention, testing schedule duration will continue to increase. By introducing continuous testing and AI/ML based tools, manual effort, and the cost of a release can be reduced, while increasing testing efficiency & quality.

ML Use cases in test planning and optimization

•  Test case identification and prioritization: Adoption of next-gen AI/ML helps identify the right set of test cases by prioritization. The prioritization is based on past defects, traceability to requirements, the number of runs among several parameters to ascertain whether a test case fails. Basis this, test case identification and prioritization give testers an objective and easy way to plan rather than relying on SME judgement.

  Test suite optimization – Sometimes, QA engineers run an entire test suite just because of minor changes within the code... Using ML tools helps define a minimum number of tests needed to check code modifications' relevance. This method can help testers identify the low-risk test cases that are unlikely to deliver any production defect. As a result, testers can focus on the most critical test cases and save efforts by running an optimized suite.

To solve the prevalent challenges in high-volume testing engagements, Cognitive Quality Engineering was conceptualized to leverage the power of AI. Mphasis’ Cognitive Quality Engineering (CQE) is an approach to implement Risk-Based Testing (RBT). CQE Platform helps clients achieve RBT transformation, making testing agile and future-proof. With advanced NLP, AI/ML, and statistical algorithms, CQE can draw insights from patterns hidden in the historical testing artifacts and past runs. CQE platform consists of several AI models, each designed to solve a unique challenge in QA effectively. CQE helps in reducing defect leakage and optimizing the testing efforts. Whether it is a time crunch or mid-sprint priorities change, the CQE platform saves cumbersome activities such as test cases identification and prioritization on a click of a button. The platform generates outputs that the test lead/manager can directly use. With scalable platform features, intuitive and configurable UI, adoption of CQE across the enterprise is extremely simple. The model helps in the reduction of planning efforts up to 75-80%. Around 40% improvement in the product coverage. Reduction in test efforts by 20% and 30% reduction in defect leakage.

While adopting AI/ML might help solve many problems, it’s not a silver bullet. Correct problem definition, measuring criteria, and availability of the right datasets are critical to the success of the CQE approach. In some cases, the current test practices and processes may need to adapt to this next-gen platform. Test cases may require some refactoring and redesigning to make them more adaptable to AI/ML. Typically, it takes few iterations of experimentation and retraining to get to best results that suit the needs of a testing team.

While Regression Test planning and optimization is usually one of the first use cases to experiment with, Mphasis CQE continually adds more AI techniques and models to solve unique problems in this space. Some of the use cases are - defect prediction, schedule prediction, smart defect classifier auto RCA, UI/UX testing models.

Our customers find CQE very useful, saving time and money while increasing product velocity. Important to note that AI/ML won’t do magic overnight, it is an iterative and experiment driven technology with the potential to unlock huge gains. Adoption of CQE will develop Quality Assurance (QA) teams into Quality Engineering (QE) capability.