Software testing software testing metrics tutorials and videos

QA Metrics That Still Mean Something in the AI Era

Software Testing Magazine — Thu, 23 Jul 2026 20:49:29 +0000

Metrics exist to answer two questions: how good is the software we deliver, and is our testing approach actually working? AI-generated tests broke the old answers to both. QA metrics were never really about numbers. They exist to answer two questions. First, how good is the software we are delivering to customers? Second, is our testing approach actually working, or are we just going through the motions? Author: Yevheniia Sakovets, https://yevheniiasakovets.com/ For years, QA leads answered both questions with the same set of proxies: how many tests we ran, how many passed, what our coverage looked like, and how many bugs we found, broken down by severity. Those proxies worked because they reflected the health of the QA process. A suite of three thousand tests meant months of real work, a stable pass rate meant the product was under control, and a falling count of critical bugs meant quality was moving in the right direction. Then AI test generation arrived and cut the link between those numbers and reality. Today one engineer with an AI tool can generate thousands of tests in an afternoon. The numbers still go up, but they answer neither question. If your quality story is still built on them, you are one hard question away from losing leadership’s trust: if AI writes all these tests, why do customers still find bugs? Why the old numbers stopped reflecting reality The test count broke first. AI floods both sides of the equation: teams receive AI-generated application code and [...]

The post QA Metrics That Still Mean Something in the AI Era first appeared on Software Testing Magazine.

Test Case Point Analysis

Software Testing Magazine — Mon, 09 May 2022 12:00:47 +0000

Software testing productivity is usually computed as the amount of testing over the effort spent for testing, but it may not be accurately measured using these size metrics. To address this issue, this article presents a new approach to estimate the software testing effort with an independent metric. The sizing method is called Test Case Point Analysis (TCPA). Test Case Point Analysis measures the size of the test case, the core item that testers create and use when performing software test execution. The size of a test case is evaluated using four elements of test case complexity, including checkpoint, precondition, data, and type of the test case. The TCPA uses test cases as input and generates the Test Case Point count for the test cases being measured. The complexity of the test case is based on four elements: checkpoint, precondition, test data, and types of test case. By measuring these four elements, this approach assumes that the complexity is centered at these elements. TCP Analysis uses a 7-step process consisting of the following stages: 1. Identify Use Cases 2. Identify Test Cases 3. Determine TCP for Test Case Generation 4. Determine TCP for Automation 5. Determine TCP for Manual Execution 6. Determine TCP for Automated Execution 7. Determine Total TCP The advantages of Test Case Point Analysis are that this software testing effort estimation metric is easy to implement and it reflects the real complexity of test cases. Furthermore, this approach is independent with the number of steps. Additional references [...]

The post Test Case Point Analysis first appeared on Software Testing Magazine.

Defining Test Automation Metrics

Software Testing Magazine — Tue, 29 Apr 2014 15:39:24 +0000

Tom DeMarco wrote “You can’t control what you cannot measure”. If test automation has always been actively discussed, the returns of automated tests were usually described in a very general way. There have been so far very few methodologies that can provide you with unbiased assessment of your software testing automation process. This article proposes some of methods to define test automation key performance indicator (KPI). Author: Dmitry Tishchenko, A1QA, http://a1qa.com The emphasis in proposed metrics is made upon two points: cost difference and duration difference Cost difference. The final costs of automated tests should be lower than those of manual software testing. If you know the frequency of builds and the economy rate, it is then possible to forecast when the efforts create value. A tester will be able to spend time to cover with manual tests more sophisticated project part. With automated tests, a testing team is able to deal with a bigger scope. The cost difference is represented by Return on investment (ROI) metrics calculated using a well-known formula: ROI = benefits/costs, where the costs are the expenses for test development, execution and support; and the benefits are the savings of replaced testing efforts. In practice, assuming that software releases are produced 4 times per week, the ROI is gained in 3-4 months if the coverage is chosen correctly. Duration difference. Testing cycle duration in automation must be shorter than in manual testing and that is why the results are gained sooner. This decreases the scope of [...]

The post Defining Test Automation Metrics first appeared on Software Testing Magazine.

How Many Test Cases?

Software Testing Magazine — Mon, 16 Dec 2013 22:10:38 +0000

Can you use the number of test cases as a metric? In this blog post, James Christie discusses this topic and the usefulness of counting test cases. His point of view is that the number of test cases and their use as a metric of progress for the software testing effort are not meaningful. What is important is how the tests run are important to validate the software under test. You can have a lot of small test cases that give limited information and a complex test case which is difficult to run but provides valuable evidence that the application is running correctly. In his experience this is however sometimes difficult to explain to management who just want reports to “manage the stakeholders”. Whether the metrics are really meaningful to assess the quality of the software is less important. Read the complete blog post on http://clarotesting.wordpress.com/2010/07/21/but-how-many-test-cases/

The post How Many Test Cases? first appeared on Software Testing Magazine.

CRAP (Change Risk Anti-Patterns) Code Metric

Software Testing Magazine — Wed, 23 Feb 2011 20:42:06 +0000

In this blog post, Alberto Savoia discusses the CRAP (Change Risk Anti-Patterns) code metric. The CRAP metric combines cyclomatic complexity and code coverage by automated tests to help identify code that might be particularly difficult to understand, test or maintain.

The post CRAP (Change Risk Anti-Patterns) Code Metric first appeared on Software Testing Magazine.

Metrics for Implementing Automated Software Testing

Software Testing Magazine — Sat, 02 Oct 2010 21:33:59 +0000

“Metrics for Implementing Automated Software Testing” presents metrics that you can use to manage the transition towards automation of software tests.

The post Metrics for Implementing Automated Software Testing first appeared on Software Testing Magazine.