A fully green traceability matrix can make it look as though requirements are properly covered. Yet a link between a requirement and a test does not prove that the test actually verifies every required behavior. Requirements coverage gaps can remain when tests are incomplete, when a requirement evolves without its tests following, or when a test changes while its traceability link remains intact. This article proposes a simple way to detect them: first understand how they arise, then structure the review by comparing explicitly required behaviors with what the linked tests actually demonstrate.
Author: Benjamin Chollet, KomAInu, https://komainu-ai.com/
Why a Green Matrix Is Not Enough
| REQUIREMENT |
| The system shall reject inputs longer than 100 characters. |
| LINKED TEST |
| Verify that the input field is displayed and accepts user input. |
There is nothing inherently wrong with the link: both artifacts concern the same feature. But the test says nothing about the 100-character limit. Traceability tells us that a relationship exists; it does not tell us what the test demonstrates.
Here, I use covered to mean that the tests demonstrate the behavior stated in the requirement. That is a stronger claim than saying that one or more tests are attached to it.
Figure 1. A traceability link can be valid even when the required behavior is not verified. (click on image to enlarge)
How Coverage Gaps Appear
Coverage gaps often hide inside tests that are otherwise relevant, useful and passing. Two causes are particularly easy to miss: incomplete verification of the required behavior and misalignment introduced when requirements or tests evolve.
1. The Test Is Relevant but Incomplete
| REQUIREMENT |
| When authentication fails three consecutive times, the system shall lock the account for 30 minutes and display a notification to the user. |
This requirement has one identifier but several obligations: the account must lock after the third consecutive failure, remain locked for 30 minutes and display a notification.
Assume the linked tests verify the lock after the third failure and the notification, but never check the 30-minute duration. The tests are still valid and the links legitimate, yet one explicitly required behavior has not been demonstrated.
The same problem appears when the right outcome is checked under the wrong conditions, when an explicit limit is never exercised, or when an assertion disappears. A test can keep executing the right scenario and keep passing without verifying the outcome we rely on it to prove. Execution is not verification.
2. The Requirement or Test Changes Without Breaking the Link
| ORIGINAL REQUIREMENT |
| The account shall be locked after three consecutive authentication failures. |
A test performs three failed attempts and checks that the account is locked. At this point, requirement and test are aligned.
| UPDATED REQUIREMENT |
| The account shall be locked for 30 minutes after three consecutive authentication failures. |
The existing test has not become invalid. It still verifies the lock, can still pass and can remain legitimately linked to the requirement. What it no longer supports is the conclusion that the current requirement is fully covered: the new duration obligation has no corresponding check.
The reverse can happen as well. An assertion can be removed or an important precondition changed in a test without altering its traceability link. In both directions, the matrix can stay green while the coverage conclusion is no longer valid.
Figure 2. A requirement can change while the test still passes and the traceability link remains intact. (click on image to enlarge)
Structure the Review to Detect These Gaps
Once the causes are clear, the review can be structured as a direct comparison between what the requirement demands and what the tests prove. Traceability then becomes the entry point for locating tests, not the evidence of coverage itself.
Step 1 – Identify What the Requirement Actually Requires
Start with the current requirement and identify the obligations it explicitly contains. One requirement can contain several behaviors without requiring a separate test for each. The goal is simply to know what must be demonstrated before drawing a coverage conclusion.
At the same time, do not turn test ideas into requirements. For “reject inputs longer than 100 characters,” testing 101 characters follows directly from the specified behavior. Testing exactly 100 characters is an obvious boundary to explore during test design, but this requirement alone does not explicitly state that an input of exactly 100 characters must be accepted. That behavior may be defined elsewhere or may require clarification.
It is therefore useful to keep three things separate: behavior explicitly required by the specification, additional scenarios worth testing, and behavior that cannot be assessed without clarification. Sometimes the right outcome of a review is not a new test, but a question for the requirement owner.
Step 2 – Read the Test, Not Just Its Link
A title, identifier or traceability link helps locate a test, but says little about what it actually verifies. For each test, examine its conditions, the action it performs and, especially, the checks it makes.
For the authentication requirement, a test might establish that two failed attempts have already occurred, perform a third failed attempt and then assert that the account is locked. Remove that assertion and the test can execute exactly the same path while no longer demonstrating the required outcome.
Step 3 – Compare Each Required Behavior With the Available Evidence
| Required behavior | What the linked tests check | Assessment |
| Account locks after the third consecutive failure | Three failed attempts are performed and the locked state is asserted | Covered |
| Lock remains effective for 30 minutes | No corresponding check found | Gap |
| Notification is displayed | The notification is asserted | Covered |
Nothing in this result says that the existing tests are poor. Two required behaviors are verified correctly. The problem appears only when those valid tests are used to support the broader statement that the entire requirement is covered.
I would describe that requirement as partially covered. Other reviews may use labels such as fully covered, linked but not covered, or unclear. These labels are useful for diagnosis, not scoring: two demonstrated behaviors out of three do not automatically make a requirement 66% covered, because the obligations may differ significantly in risk or importance.
Step 4 – Correct the Cause of the Gap
Once a gap is visible, the corrective action depends on its cause. A new test may be needed, but an existing test may simply need a stronger assertion or a missing precondition. An outdated test may need revision, and the requirement itself may need clarification.
A Green Matrix Is the Beginning of the Review, Not Its Conclusion
Traceability remains valuable: it shows which requirements are associated with tests and makes requirements with no recorded test links easy to identify. What it cannot establish by itself is whether those tests still demonstrate everything required today.
For a small requirement set, this comparison can be performed manually. At larger scale, keeping it current becomes harder because requirements and tests both evolve. The core question nevertheless stays the same: which current test demonstrates each behavior that the current requirement actually demands?
Once the traceability matrix is green, one question still remains: what does each of those tests actually prove?
Author Bio
I’m Benjamin Chollet, Co-Founder and CPO of KomAInu, with a background in critical software engineering and verification and validation (V&V). My work focuses in particular on traceability, requirements coverage and improving verification activities in critical software environments.
Leave a Reply