When you add new code to your applications, it is associated to the test scripts that should validate it. Do you give enough importance to these tests? In this article, Eli Lopian explains that when pull requests are opened, you should review the tests scripts as much as the code.
Author: Eli Lopian, Typemock, https://www.typemock.com/
Modern software development has become remarkably disciplined about reviewing production code.
A pull request is opened. Developers examine the implementation, question architectural decisions, look for security issues, consider maintainability, and discuss whether the change belongs in the codebase at all. Automated checks then add another layer of protection through compilation, static analysis, security scanning, and continuous integration.
But there is an important gap in this process.
The tests added by the pull request often receive far less scrutiny than the production code they are supposed to validate.
A developer may submit hundreds of lines of implementation code alongside dozens of new unit tests. The production code receives detailed comments. The tests are compiled and executed. If everything turns green, the pull request moves toward approval.
That process assumes something that should not be assumed:
A passing test is not necessarily a good test.
As AI-assisted development makes it dramatically easier to generate automated tests, this distinction is becoming more important.
A Passing Test Can Still Be a Bad Test
Continuous integration answers an essential question:
Does the test currently pass?
It does not necessarily answer:
Does this test provide meaningful additional confidence in the software?
Consider a pull request that adds 30 unit tests. All 30 pass. The build is green, code coverage has increased, and the change appears successful.
But perhaps ten of those tests verify behavior already covered by existing tests.
Perhaps several depend on implementation details rather than observable behavior.
Perhaps some contain complicated mock configurations that make them fragile.
Perhaps others interact indirectly with the filesystem, environment, clock, or network, making them less deterministic than their names suggest.
The suite has grown, but its quality may not have improved proportionally.
This is the difference between test execution and test review.
Execution tells us what happened when the tests ran.
Review asks whether the tests were worth having in the first place.
Tests Are Production Code Too
There is an obvious reason production code receives so much attention during code review: it is software.
It can contain defects. It can introduce unnecessary complexity. It can duplicate functionality. It can create dependencies that make future changes harder.
The same is true of test code.
Tests contain logic, setup, dependencies, abstractions, mocks, assertions, and maintenance requirements. They can become complicated and expensive just like production code.
Yet organizations frequently apply different standards to the two.
Imagine a developer submitting 500 lines of production code with no meaningful review. Most teams would consider that unacceptable.
Now imagine a pull request adding 500 lines of automated tests. In many organizations, the reaction is considerably different.
If the tests pass, they may receive only a superficial inspection.
There is little technical justification for treating the two differently.
A test is still code.
And code can be wrong even when it executes successfully.
What Does a Test Review Actually Look Like?
Reviewing tests does not mean checking whether every assertion uses the preferred syntax.
The more important questions are about value, isolation, intent, and maintainability.
For example:
Does the test verify behavior that matters?
A test that duplicates an existing scenario may increase the size of the suite without increasing confidence.
Does it test a distinct behavior?
A large collection of nearly identical tests can create the appearance of extensive coverage while providing relatively little additional protection.
Is the test isolated?
A unit test that unexpectedly depends on the filesystem, network, environment variables, system time, or external services can become fragile and difficult to diagnose.
Are mocks and fakes actually necessary?
Mocking frameworks are useful tools, but unnecessary mocking can make tests harder to understand. A test containing extensive mock configuration may be testing the test infrastructure as much as the production behavior.
Does the test communicate intent?
A good test should help another developer understand what behavior the system is expected to provide. If the reader must navigate through layers of setup before discovering the actual behavior being tested, the test may need simplification.
These are review questions, not execution questions.
A CI server cannot necessarily answer them simply by running the test.
Coverage Is Not the Same as Confidence
Code coverage has become an important metric in software engineering, but it is frequently misunderstood.
Increasing coverage can be useful. It can identify areas of the application that receive little or no automated testing.
But coverage measures what code was exercised, not necessarily whether the tests meaningfully protect that code.
A test can execute a line without verifying an important outcome.
Multiple tests can exercise the same path without providing substantially different protection.
A pull request can therefore increase coverage while simultaneously making the test suite more difficult to maintain.
This creates an important distinction:
More tests are not automatically better tests.
The objective should be meaningful confidence, not simply a larger number of test cases.
AI Changes the Economics of Test Creation
The argument for test review becomes stronger with the increasing adoption of AI-assisted development.
Generating tests used to require significant developer time. A developer had to identify scenarios, write setup code, create mocks, formulate assertions, and maintain the resulting tests.
AI coding assistants can now produce many of those tests almost instantly.
That creates an enormous productivity opportunity.
It also changes the bottleneck.
When writing tests is expensive, teams naturally limit the number of tests they create.
When generating tests becomes cheap, the limiting factor becomes determining which tests are useful.
A developer can ask an AI coding assistant to generate tests for a method and receive dozens of cases. Some may be valuable. Some may overlap with existing tests. Some may be excessively coupled to implementation details. Others may add little meaningful protection.
The problem is no longer simply:
How can we create more tests?
It is:
How can we determine which tests deserve to remain in the test suite?
That is a quality problem.
The Test Suite Has a Cost
Automated tests are often described primarily as an investment.
They are.
But they are also a long-term liability if they are poorly designed.
Every test has a maintenance cost.
When production behavior changes, tests may need to change with it. When dependencies evolve, mocks may break. When execution time increases, CI pipelines become slower. When a test becomes flaky, developers must investigate it.
At sufficient scale, these costs become significant.
A suite containing 100,000 tests is not automatically healthier than one containing 50,000.
The relevant question is how much useful confidence those tests provide relative to their cost.
This suggests a useful way to think about test quality:
The goal is not to maximize the number of tests. The goal is to maximize valuable coverage and confidence while minimizing unnecessary complexity and maintenance.
Test Review Should Become Part of the Pull Request
Most mature development organizations already have several automated quality gates.
A typical pull request may go through compilation, static analysis, security checks, unit tests, integration tests, and human code review.
Test review should become another part of that process.
The workflow could be straightforward.
When a pull request introduces or modifies tests, the system could analyze the changes and identify potential concerns.
Are new tests substantially similar to existing tests?
Do multiple tests appear to exercise the same behavior?
Are tests dependent on potentially unstable external state?
Are there unusually complicated mocks?
Are assertions weak or missing?
Do the new tests primarily verify implementation details?
Are there patterns associated with flaky or expensive tests?
The purpose would not be to automatically reject tests.
It would be to give reviewers better information.
A reviewer could then spend time on the cases that actually deserve human judgment.
Automation Should Assist Reviewers, Not Replace Them
There is an important distinction between automated test analysis and automated test approval.
Software engineering still requires human judgment.
A tool may identify that two tests appear redundant. A developer may know that the tests intentionally represent two different business scenarios.
An analyzer may flag a mock as unnecessary. The developer may have a specific architectural reason for using it.
Automation can identify patterns and risks.
Humans determine intent.
That makes test-review automation similar to other quality tools already accepted in software development. Static analyzers do not replace code review. Security scanners do not replace security engineers. Linters do not replace developers.
They make human review more efficient.
The same principle can apply to tests.
Test Review Could Become a New Quality Gate
The software industry has spent decades developing increasingly sophisticated approaches to production-code quality.
We use version control, pull requests, peer review, automated testing, static analysis, dependency scanning, security analysis, and continuous integration.
The next evolution does not necessarily require another way to execute tests.
It requires a better way to evaluate the tests themselves.
A mature pull request process could therefore look something like this:
Production code: Does the implementation work and meet engineering standards?
Automated tests: Does the software behave as expected?
Test review: Do the new and modified tests provide meaningful, maintainable confidence?
Those are three different questions.
Treating them as one is what creates the current gap.
The AI-Generated Test Problem Will Only Grow
The industry is moving toward development environments in which generating code becomes increasingly inexpensive.
That includes test code.
If developers can generate 50 tests in the time it previously took to write five, organizations will inevitably accumulate more automated tests.
That is not necessarily a problem.
The problem occurs when test-suite growth outpaces our ability to evaluate test quality.
Without review, AI can accelerate not only the creation of useful tests but also the creation of redundant, fragile, or low-value tests.
The answer should not be to stop generating tests.
It should be to introduce quality controls around generated tests.
AI can help write the tests.
Other automated analysis can help evaluate them.
Developers can make the final decision.
From Test Quantity to Test Quality
For many years, discussions about testing have focused on questions such as:
How many tests do we have?
What percentage of the code is covered?
How quickly do the tests run?
How many tests are failing?
Those metrics remain useful.
But modern development requires another question:
How much value does each test provide?
That question becomes particularly important as AI changes the economics of software development.
When code is scarce, optimization focuses on producing more code.
When code becomes abundant, optimization shifts toward determining which code should exist.
Tests are no different.
The future of automated testing will not simply be about generating more tests. It will be about maintaining a test suite in which every important test has a clear purpose.
The Pull Request of the Future
The traditional pull request asks reviewers to evaluate the production changes and then confirms that the automated tests pass.
A more mature pull request will ask reviewers to evaluate three things:
What changed?
Does it work?
Are the tests that validate it worth keeping?
That final question deserves considerably more attention than it receives today.
The software industry has already accepted that production code should not enter a codebase without appropriate scrutiny.
Automated tests should receive the same treatment.
Especially now, when AI can generate them faster than humans can reasonably review them.
The next step in software quality is therefore not simply writing more tests.
It is learning to review them.
Because a green build tells us that the tests passed.
It does not tell us that the tests were good.
About the Author
Eli Lopian is founder and CEO of Typemock, a provider of AI-powered unit testing and code quality solutions for .NET and C++. With more than 20 years of software engineering leadership, he specializes in unit testing, Agile, TDD, and AI-assisted development. He is also the author of AIcracy: Beyond Democracy.
Leave a Reply