The first part of this article presented some of the current challenges of performance testing. It discussed also the data and times pillar of performance testing. This second part covers the resource and cost aspects. The author shares some final thoughts on the future of performance testing.
Author: V.M.Guruprasath (AIS,MCTS)
Resource
Performance testing is a niche skill and availability of expert resources in the market is very low. Today’s trend has turned to be that if there is a word “Performance testing” in his/her resume, the resource gets deployed to any performance testing project. Your need to understand that, like SAP developers are different from Java developers whom we cannot interchangeably utilize for any development projects; similarly, a performance test engineer expert in MicroFocus Load Runner’s HTTP/HTML protocol cannot be an expert of MicroFocus Load Runner’s SAP GUI protocol. Though it is the same tool, the functions used and the way the performance testing scripts need to be developed are totally different. It takes time, effort and experience to build expertise in a single protocol. This might range from one week, one month or one year, depending on the resources’ independent skill level and the level of training.
On top of this, we have multiple tools like JMeter, VSTS, etc., each having their own language for script development. As an example, MicroFocus LoadRunner tool uses modified ANSI-C for script development (mostly), whereas JMeter uses JavaScript. These are two completely different languages.
A performance testing team member working on a project on a tool and a protocol, will get less time to explore other tools and protocols. When the resource rolls off one project, if a new project starts that needs performance testing on a different protocol or tool, he might not be the ideal candidate as he needs to be provided with training. Unlike functional testing, there is a technical learning curve and it takes more time, which every management need to understand.
Before, there used to be time allocated in the overall project timelines for team to get trained on the application functionality and bridge any technical tool gaps. Some budget was allocated for the team’s training. Today, this budget has eroded in most of the projects for some reasons. Organizations need to understand this technical need and should also work on planning upskilling teams before deployment to new projects that have new protocol, technology or functionality for the performance testing.
With the cost-cutting measures in place, coupled with upstream delays, performance engineers get on boarded quite late in to system.
With less time to go-live; while his data requirements mentioned under data section are unavailable or only partially available; with functionality, protocol and tool expertise constraints explained in the above paragraph that need to be addressed by performance testing team in short time, all this forces the performance testing team to bridge multiple gaps in a very short time. This leads to a false notion that the team is not expert enough and performance testing is a bottleneck. However, this is just the result of the lack of data, time and resources that need to be guaranteed for the performance testing team.
Moreover, performance testing is a collaborative effort that requires support from almost every team in the application hierarchy. They need support of solution architect, developer, environment, infrastructure, business analysts, database team, middleware, functional testing team, automation and test data management team for effective planning, designing, execution and result analysis. The absence of effort estimation from the above teams to support performance testing in their estimates, leads to lack of data and support for the performance testing activities from the supporting teams which further aggravates the bottlenecks for performance testing.
Cost
Performance testing is a costly affair. Any attempt to portray performance testing as a low-cost affair is a farce. When compared with the costs associated with the lack of performance in the production, the preventive measure of investing in performance testing is beneficial. There have been cases of absence of performance in production, leading to the entire project being scrapped and all the money went down in the drain.
The huge data requirements, the time that is required, the resource constraints and the tools ultimately contribute to the cost. One of the new trends is absence of production-like performance testing environment for performance testing which was a de facto standard a decade ago. Performance testing is not only about how quickly the application code responds. We need to understand the response times are impacted by many factors like bandwidth allocation, configuration of various servers, log levels, database index and so-on that can be effectively tested only on a production-like environment.
There is a common wrong perception that performance test execution is a click of a button and why we need so much money?
The following analogy would help us to clear the air. Let us take the example of a flight, when the flight has reached its altitude, all the pilot needs to do, is click on “Auto-pilot”. Isn’t it?
But for the pilot to click the “Auto-pilot” button, we need to have the pilots trained (we have a minimum of 2 pilots), the airport infrastructure, the check-in formalities, validating & loading the right passengers to the flight, ATC clearance for the right runway, taxing on the runaway, slowly accelerating, take-off and reaching the desired altitude. After reaching the desired altitude, yes, it is click of a button.
Similarly, for clicking on the “Run” button in Performance testing, we need huge efforts in terms of equipping the resources with training and tool sets, support from different teams, building the scripts, test data loaded and ready, environment clearance for the right environment, smoke tests, setting up the scenario for the slow ramping up of the load, steady state and ramp down. After setting up the scenario, yes, it is indeed a click of a button.
We also need to understand that unlike functional testing where when the automation scripts are ready most of the efforts are done, this is not the case for performance testing. Here completion of the scripts is only half of the overall efforts. We have execution and result analysis to complete which as well are equivalently resource consuming.
The Solution
The table given below summarizes the four pillars of performance testing, their current state and the possible solution:
Conclusion
As the ancient Indian text Subhashitavali rightly says: “The wise could never launch anything which bears no fruit, which ends tragically, which has no gain over the expenditure and which is impossible”
The bolstering of the above four pillars of performance testing helps to bear the right fruits for the investments over the expenditures and makes Performance testing efficiently possible. It will also set a strong base for performance testing to face any upcoming disruptions.
The performance testing team also need to get themselves geared for enhancements and upskilling. Enhancements like leveraging Artificial Intelligence for performance trend analysis and upskilling themselves to the latest technologies. For example, learning new protocols like Constrained Application Protocol (CoAP) and MQ Telemetry Transport (MQTT) for Internet of Things (IoT) and the like.
The future is no longer on knowledge and experience. It is on creativity and the performance testing team need to be as well creative to assimilate and upskill the new disruptions to complement Performance testing.
Finally, would like to conclude stating that the presence of a medical doctor is necessary as long as human beings are on the planet, despite the advances in the medical field; similarly performance testers are needed as long software exists despite the disruptions, provided the four fundamental pillars discussed here are rightly understood and bolstered.
About the Author
V.M.Guruprasath works as a Senior Manager in an organization which is a global leader in consulting, technology services and digital transformation. Guru has over 14 years of experience in performance testing & delivery management for multiple domains ranging from large multi-national financial organizations like credit bureaus, credit card corporations, Insurance & banking institutions, to retail and construction businesses. Learn more about V.M.Guruprasath on his linkedin page: https://www.linkedin.com/in/vmguruprasath/
Disclaimer: Views expressed in this article are of the author as individual and does not reflect the view of any organization