Performance Testing for a SaaS based B2C Application
Experion worked with the client as a technology partner to develop and deploy the web and mobile based aggregator platform. End users of the solution include those who plan to organize a house party or wedding reception or similar events, and they can make a request to catering service providers to organize the event through the application.
Apart from providing aggregator services, the client also organizes food festivals, which also acts a means of advertisement and marketing. The activity involves the client bringing in caterers and end users on to a single stage, where caterers put independent stalls to sell their delicacies.
The client uses e-tickets to charge end users for the purchases they make. End users can purchase e-tickets for specific amounts through the ticket module in the mobile application. A QR code which holds the purchase value of the ticket is issued to the end user. They are given the option to use either the soft copy or printed copy of the QR code saved in their mobile phones. The QR code takes the role of currency and acts as the medium for making payment transactions for the services availed in the event. Service providers use QR code scanner to scan and deduct the purchase amount.
Before rolling out the solution into real world, various performance parameters had to be measured and validated, as it was critical that the solution scaled gracefully on going live, without any service disruption due to performance degradation. Client wanted to make sure that the application would be able to withstand the planned capacity for an event and also to evaluate whether the technology infrastructure is adequate for smooth operation.
Performance Testing
Experion has an ‘Independent Verification and Validation Team’ which conducts functional as well as nonfunctional testing such as security testing, compatibility testing, performance testing, etc. for web/ mobile applications. This team was assigned to conduct performance testing to ensure that the application runs smoothly with the provisioned infrastructure. Extensive tests were run on the QR Code scan & redeem functionality and the transaction response was investigated for different scenarios. The test was designed to expose any potential performance flaws with respect to the provisioned infrastructure as well as the application architecture. Performance testing was chosen to bring clarity on the following aspects:
- Ensure non-functional constraints such as reliability and responsiveness match expectations under real-world conditions
- Ensure error percentage did not exceed the accepted threshold level
- Validate that the response time for APIs involved in the scan and redeem functionality did not exceed the accepted threshold
- Benchmark the capacity of provisioned cloud instance during the load, to help infrastructure managers make smart decisions while scaling up
Approach adopted for testing
After a thorough study and research of business workflow model, Experion testing team arrived at a projected number of service providers and simulated a virtual food festival event scenario. The team studied the flow of the application and identified the steps involved in scanning & redemption of the e-tickets. Based on that, the test was modelled. Team opted to engage a reliable, performance testing Platform-as-a-Service (PaaS) solution, BlazeMeter, to work as the load generator on application servers. Request response time along with server KPIs were measured as part of the performance testing process.
Figure 1: Test Configuration
In the above diagram, the vertical flow shows the spawn of virtual users and the horizontal flow shows the sequence in which APIs are invoked in the test configuration. BlazeMeter was configured to spawn virtual users to match the projected number of service providers, in separate threads which run concurrently/ parallelly. In each thread the APIs under test are invoked in sequential order; Login API followed by Scan API and then Redeem API, which represented the Scan and Redeem work flow. A think time or a delay of 5 seconds between each Scan and Redeem routine was applied to model the flow as close to reality as possible. The test was conducted for 20 min and this was considered as a Run. After every Run a considerable delay (pacing) of almost 10mins was given before starting the next run. Results of best of three runs were considered for analysis.
Test Results
Experion testing team analysed the data collected to benchmark threshold values of the server. The results showed that the infrastructure provisioned for the server is adequate. In addition to that, the load was seen evenly distributed among the application servers behind the load balancer, indicating configuration setup for the load balancer is correct for the requirement. Separate test was done to confirm the configuration setup for autoscaling facility available on the cloud. This gave immense confidence to the client that in case of a peak load during a food festival which caters thousands of food lovers and caterers, the infrastructure and the application will deliver seamlessly.
During the time of performance testing, the application servers are already under sufficient stress. At this time, testing team at Experion was conducting exploratory manual testing to verify the application for any performance flaws. List of mobile devices which could be used by the caterers was shortlisted. The manual testing was conducted on these set of devices or like devices to mitigate risk when devices of multiple models/ makes are used during a festival.
The test results of the performance testing were an assurance to the client to face any festival with a sizeable attendance. The request response time was found to be within the acceptable limit. As part of the engagement, application logs were scrutinized, and database was queried to check integrity. The results were positive.