Cloud Based Performance Testing Strategy

Performance testing strategy on cloud can be broadly categories in two ways:

• When whole solution / application shifts on cloud and testing is performed using cloud services
• When application is in-house and performance lab is shifted to cloud to generate load traffic

The scope of this article revolves around the second cloud based testing approach where the application resides within the on-premise lab, and traffic is being generated using cloud computing tools and services available in the market. Below architecture shows how the Infrastructure as a service (IaaS) model has been utilized to leverage the cloud services for traditional IT testing process.

WHEN TO SHIFT TO CLOUD

• High user load requirement, and current infrastructure is not dynamic enough to cater the load in short period of time
• Customer facing applications, hosted outside the firewall
• Application hosted in various geo locations, to simulate similar kind of scenario
• Performance lab needs to be setup quickly for production testing
• Performance lab being setup for the first time and requires OnDemand resources
• Hardware maintenance cost needs to be optimized

HOW TO STRATEGIES

As a necessity, cloud setups are considered to provide elasticity and flexibility along with quick adoption towards desired configurations of components, whenever and whatever required, based on various factors. Hence, for evaluation purpose, we have chosen a setup to suit business requirements. The key areas to consider are mentioned under Pre-requisite, Test Scripting, Test Execution and Reporting sections.

PRE-REQUISITE FOR CLOUD TEST LAB SETUP

• Environment Setup Complexity - The time needed to build a setup to conduct performance testing, evaluate current infrastructure based on the latest application and check for environment readiness, could be a game changer in selecting the sprint wise model. Variety of configurations suitable to the situations could also be evaluated in the pre-scripting phase.
• Data Check & Retention - Cloud tools offer data retention policy based on the enrolled package. If there is any critical data, such as banking or regulatory related information, ensure to encrypt the data, self-clean the logs or archive the logs in private domain, or opt for virtual private cloud.
• Test Data Setup - It is easy to plugin applications in cloud-based setup to collate the information such as address, SSN and trade transactions, which help in evaluating the performance of these applications in real world. The data required to evaluate can eventually be used with obligatory tinkering for performance test benchmark.
• Workload profile - Prepare a workload model with different types of performance test, the rounds of execution for each test and the user load to be distributed for your application. Deciding the user load profile beforehand helps choosing the cloud-based tool and the type of license to procure.
• Monitoring - Consider the cloud tools that offer inbuilt monitoring or provide integration with the in-house existing monitoring tools like AppDynamics/ Dynatrace/ Newrelic, to understand how the application is performing while using cloud machines.
• Firewall Readiness - Having the required firewall in place while executing the scenario on cloud performance lab saves time.
• Cloud Tool Integration - To perform various checks from the engineering aspect such as a memory dump analyzer or a network virtualization service, you can integrate the cloud based performance testing tools with monitoring, logging or reporting tool available in market or at own premise. The cloud tool also works well with Jenkins/Bamboo for CI system integration.
• User\Project Management - Cloud tools can be used by different teams having personalized concurrent user license.
• Security Check - Check if ‘in premise’ setup for banking applications or publicly exposed domain applications are firewall enabled, and there is no performance drop with or without firewall enablement.
• End User Experience - End user or client-side performance evaluations becomes a more prominent factor whenever an application is UI centric or rendered heavily, based on user inputs.

TEST SCRIPTING

• Building scripts on cloud - Cloud tools such as LoadRunner Cloud, LoadNinja etc. create scripts on the fly and provide ability of self-enhancing the scripts. For example: In LoadRunner cloud, you can use CSV file containing list of REST API endpoints to generate API scripts. You can choose HAR upload option or create on cloud scripts using browser record options & playback.
• Modify on the Fly - TTraditional setup of scripts is a tedious job, considering the changes, version maintenance, data insertion & reuploading. However, with cloud-based tools, check-in and check-out is easy and faster.
• Create on premise Scripts - With cloud tool, you can choose the traditional method of scripting on your own machine and upload.

TEST EXECUTION

• Upload scripts - Depending upon the cloud tool chosen, upload the scripts for your run.
• Geo-location-based scenario - Plan your scenario to use various geo location and network virtualization techniques provided by cloud tools.
• Get evaluations on the go - Choose the cloud based performance tool which benefits you providing additional monitoring factors such as on the fly comparison between two load tests, auto reporting of any anomalies, setting triggers and alarms for any bottleneck/issue. Whenever required, if we can derive a matrix evaluation of current application performance based on existing as well as target environment availability should also be journeyed.

REPORTING

• Report Ease - You can limit the members accessing your reports based upon the provided authorization. Set template to auto generated report available on the go.

Key factors to consider for Cloud-based testing strategy

TYPE OF APPLICATIONS

The public facing application such as Web or Rich UI based interface, where different users access the application globally through the web firewall, is considered as a better candidate for cloud-based testing as compared to intranet applications which access by limited set of users.

TOOLS SELECTION

There are several cloud-based testing tools available in the market to fulfill the performance test requirements. Select the tools based on business requirements, ease of scripting and the number of protocols supported, the future needs related to continuous product development requirements and reusability aspects.

In case lot of endurance tests are being run (more than 72 hours tests),opting for packaged pricing will work better. High level tools analysis will help to give quick glance of the tools and performance testing application requirement they support.

One can pay per use for the cloud tools, however they are packaged priced in terms of -

• Credits
• Subscription
• Virtual User Hour/Days

OTHER FACTORS TO CONSIDER

• Firewall restrictions -
It depends on the type of performance lab architecture one has. If you are taking your controller to cloud, there is an ease for cloud LGs to communicate with your controller; no need to cross the firewall to connect with this LG. But if controller wants to connect with a monitoring tool installed on your application server, which is available in your domain, you need to open another firewall port for this communication.
Similarly, the cloud LGs have restrictions on communicating with your AUT behind the firewall. This way you end up exposing a lot of firewall ports for the testing purpose, which goes through endless approvals for enabling the ports.
• Data privacy -
Testing a banking financial or regulatory application, where you are worried about data being sent to the outside network, is a challenge because cloud is a multitenant system. A single network interface firewall is virtualized and shared between different clients, and any data transferred over the cloud network will be exposed to outsiders.
Test data should always be password protected & Personally Identifiable Information (PII) must be encrypted. Data replication from production needs to follow a strategy for data morphing, etc. before it is being used for testing.

Mitigation: One can opt for virtual private network & use virtual private cloud (VPC) to prevent exposing the personal data to outside world, with data encryption/ morphing strategy in place.

• Security -
Cloud computing emphasizes on security and follows “Mutual Shared Responsibility Model”, which means security is the responsibility of provider as well as the user. The supplier ensures security of their infrastructure keeping the client application & data protected. On the other side, the user must safeguard their exposed application / application data through timely checks and strong authentication measures to encrypt password or other sensitive data such as Personally Identifiable Information (PII), specially when setting up on public cloud.
Alternatively, one can adopt VPC to ensure better security, but again subjected to user adherence upon security rules & policies.
• Varying Application Performance -
At times, you don’t get the same Load Generator (LG) in same health (resource utilization) as different LGs may be provisioned for each test run (not in case, where you have dedicated cloud resources). You share a network interface card among other systems and the network bandwidth is not always constant across the testing intervals/days. Cloud LGs spin in programmatic manner and there is no guarantee that your machine /LG would get the same amount of CPU or RAM. You might not know under how much load the actual physical machine already is. So, the performance of the virtual machines under different runs can differ. For instance: the 1st run can give a response time of 1 sec and the 2nd run can give 1.2 sec. The difference would vary from milliseconds to seconds, depending upon the acceptance criteria for your application. If you have trading application, even 0.1 or 0.2 sec of delay can cause a considerable impact.

Mitigation: Repeat the number of tests, take the 90th, or 95th or 97th percentile and calculate the mean deviations between these runs.

Benefit of Cloud-based testing

COST SAVING

Cloud capability of pay-as-you-go model gives flexbility of having testing environment available on-demand, which in turn results in huge savings on software licenses, hardware procurement and IT support team maintence costs. Peformance testing team get the flexibility to simulate various performance test scenarios by selecting appropriate infrastructure setup , tools & lincense etc. at ease, to perfom the tests without worrying about the project cost.

REAL USER EXPERIENCE

Exposing applications to global customers calls for good performance testing strategy to test latency & other load factors . Cloud based testing enables user load distribution from different geographical lcoations, to simulate real user experience. With this setup, peformance testing team can check the fail-over capability of the servers and network architecture monitoring by enabling network virtualization option; effectiveness of application code can also be checked for features like concurrency, caching, cookie, session, dabase lock scenarios etc.

Mimicing the real world environment offers the best results, which is not possible to simulate within the own premise of performance test lab setup.

INFRASTRUCTURE EASINESS

Peformance testing is not able to meet on demand requirements that require heavy load testing, which in turn needs additional hardware to support, different geographies to test, and so on. In-premise infrastructure always has dependacy on IT support team and upgrading existing setup is always time cosuming and expensive. Cloud infrastructure is able to meet unanticipated demands easily by allowing quick and easy access to hardware & tools, eventually saving cost and ensuring customer satification.

SCALING

Cloud offers flexibility of infrastructure with the help of single click, scaling up or down the computing resources as per business requirements and monitoring key performance indicators at various load conditions for application under test. Even performance testing can easily be carried out for extreme load test scenarios.

AVAILABILITY

Performance testing lab and resources are available 24/7, and even the lab setup can be accessed from anywhere and anytime. This gives flexibility to the testing team to schedule tests as needed, without any dependency on IT support team. On cloud, the applications can be tested on the fly, as per the requirements, with less pre-requisite/checks performed, as compared to in-premise test lab. This way testing team can invest more time on test analysis and debugging.

Conclusion

Intranet applications (that resides inside the firewall) do not have public facing scenarios or heavy traffic can be tested within on-premise performance test lab. However, testing customer /consumer facing application exposed to global customer or users , requires to be tested for real-world scenarios are recommended to test on cloud based performance testing lab.

Setting up on premise testing lab and test applications on this set up require time, resources and efforts to provision the system to generate the on demand virtual user load. Usually on premise testing lab has limitation on the number of the virtual users and geo location load that traditional license tools can generate within its limited infrastuture capacity. If a sudden requirement comes up to generate additional traffic, user loads etc., there is no way to dynamically scale up the user load beyond what has already been provisioned.

Cloud based peformance testing lab can leverage the platform efficiently, enabling enterprises to save infrastructure/hardware cost. This is because running critical tests that demand high traffic is a periodic activity rather than continuous one, and thus investing over other learning curves can help organizations to follow Shift Left approach.

It is important to understand how licensing works for a cloud tool, how we pay the vendor for using their services, how costs are going to be affected as you attempt to test for more users, whether to set virtual private cloud for business-critical data load testing, and so on. Prior to moving for cloud-based performance testing approach, users should consider the above pointers and strategically plan performance testing considering the illustrated factors described in this document.

Cloud based Load testing Tools

References

https://www.cigniti.com/blog/continuous-performance-engineering-agile-devops/

Performance Testing in Cloud Right Before Your Eyes Webinar Series | Cigniti Technologies

https://www.softwaretestinghelp.com/cloud-performance-testing/

And LVMH is not alone. Scores of companies across industries distinguished themselves by recognizing the need to switch tracks. These organizations equipped the workforce with the skills, tools and mental approach needed to stay relevant in changing times by providing in-demand goods and services.

Expanding Brand Influence In Extraordinary Times

San Francisco-based Eat Just, which is known for creating a food system where everyone eats "well," continues to operate through the pandemic and continuously looks for opportunities to serve others, as well.

One of the ways it has expanded its role is by partnering with Beyond Meat — a Los Angeles-based producer of plant-based meat substitutes — to provide meals to essential workers across southern California. Eat Just has also stepped up to supply its products to Support + Feed restaurants, an initiative seeking to provide nutritious, plant-based meals to hospital workers, shelters and senior care centers.

Again, Eat Just has not just decided to give up what it does. It recognizes the need to step out of its comfort zone and serve communities and groups in need of attention in extraordinary times by leveraging its core strengths in manufacturing and distributing nutritious, nonmeat protein food.

Reimagining Business Value And Identity

Next, let's look at another disruptor: the $230 billion-a-year convenience store industry goPuff. The Philadelphia-based company started by two college students to create easy access to "late-night cravings" continues functioning during the pandemic and has even hired new drivers to meet an upsurge in demand for items such as cleaning supplies, toilet paper and essential medicines. The company, which offers an app for users to place orders and have items delivered, has even joined forces with a distillery to both source and distribute hand sanitizer.

Further, to help customers feel safe, goPuff launched no-contact delivery as an option on its app so orders can be securely dropped off on customers' doorsteps. Additionally, the company has committed to delivering digital credits worth $1 million to hospital staff as part of its health care support initiative. Under this initiative, goPuff will provide delivery of essentials, including cleaning supplies, over-the-counter medications, food and drinks, to the homes of health care professionals so they can focus on their core work: caregiving.

The company, known in over 500 U.S. cities for working with distributors to deliver personal-care items, snacks, drinks and over-the-counter medicine for a low, flat fee, rose to the occasion. It sourced a high-demand product during the pandemic and provided fast delivery of services to groups and communities on the frontlines of the crisis.

Once again, it is important to underline that the online convenience store's pivot does not need to remain a central feature of its business model once the pandemic is over. What is valuable to note is the way the company has continued its everyday business while expanding its remit to target a group whose need for essentials was expedited and provided free of cost.

This brings me to another related theme. The pandemic has accelerated a shift in business mindset to the need for companies to be concerned about the welfare of their stakeholders and the planet. It may take more time for us to know to what extent our modern lifestyle is to blame for the emergence of the novel coronavirus, but businesses of the future will no longer be able to hold their own without demonstrating an ability to foster connection, care and sustainability.