A recent survey from the Methods & Tools software development magazine shows that the load testing activity is performed only in a minority of software development organizations. The poll asked the question: “Does your organization perform load / performance testing?”
Number of respondents: 217
End of survey: July 2014
Despite the growing adoption of open source load testing tools like JMeter and the large availability of free load testing services on the web, this survey reveals that load testing is still considered as an optional step in the software development lifecycle by two-third of the respondents. The consequences of inadequate load testing has been recently made more visible with the big failure of the Healthcare.gov US web site that crashed during its launch process. This issue was widely attributed to a lack of proper load testing.
These results are in the same range than what was found in similar surveys. The Software Test in Practice survey [1] found that 47% of projects implemented load/performance testing and 21% were not using it. The State of Testing Survey 2013 [2] also has an implied result of 40% of respondents’ organizations that practice load testing. In the same survey, 41% of respondents defined the load testing skill as “very important”, 52% “important” and 7% “not important”. The results for the same question about functional testing automation skills were 53%, 40%, and 7%. In a 2014 survey about mobile testing [3] the availability of performance metrics was rated 7,3 on a scale of ten and that users expect their mobile app to be responsive in 3 seconds or less. In the World Quality Report 2013-14 [4], 54% of participants said that they pay special attention to performance requirements and risks. Only 31% do the same when they migrate their applications to the Cloud, but 38% pay special attention to peak load requirements and risks.
All these numbers show that load testing could still be a “second-tier” practice in the software development life cycle, as software testing itself is already a phase that you mostly do if there is still time before the project delivery deadline. Software developers think about it mainly when their software is accessed by external customers or when they rely on external resources, like traditional or cloud hosting providers, for their software infrastructure.
Resources
[1] Survey 2011:Software Test in Practice (PDF)
[2] State of Testing Survey 2013 (PDF)
[3] The State of Mobile Software Quality 2014 (PDF)
[4] World Quality Report 2013-14 (PDF)