Certainty Isn't Correctness: The Cost of Trusting AI-Written Code

AI-written code can sail straight through your CI pipeline — lint clean, tests green, coverage above threshold — and still ship a bug nobody thought to test for. This article breaks down why AI code passes Continuous Integration (CI) checks built for a different generation of bugs, and lays out a short list of concrete, budget-free gates that catch what your CI pipeline can’t see today.

Author: Suprajyotsna Dasari

A few months ago, on a project piloting an AI-built pull-request reviewer, I watched a change clear our Continuous Integration (CI) pipeline in under four minutes. Lint was clean. Unit tests were green. Coverage sat above our threshold. The dashboard even threw us the little green confetti it saves for merges it’s proud of. It looked exactly like the hundred safe merges before it. Two days later, a downstream dashboard started showing physically impossible temperature values for one specific tag — its raw-to-engineering-unit scaling had silently flipped from Celsius to Fahrenheit under one code path — a case nobody, human or AI, had thought to write a test for.

Nothing about that Continuous Integration pipeline lied to us. It answered every question we had told it to ask. We just hadn’t updated the questions for the kind of mistakes AI-generated code actually makes. That gap — between what a green build proves and what we assume it proves — is the real problem with AI-assisted development right now. It’s also a fixable one, and fixing it doesn’t require new tooling budget, just a different set of gates.

The gap nobody’s CI pipeline measures

A controlled study published in 2023 gave one group of developers access to an AI coding assistant and left another group without it, then had both groups complete the same security-relevant programming tasks. The AI-assisted group wrote measurably less secure code on most of the tasks. The part worth sitting with is the second finding: the AI-assisted group was also more confident their code was safe than the unassisted group was. Confidence went up exactly as correctness went down.

A separate randomized trial in 2025 found the same pattern in productivity instead of security. Experienced developers working in their own large, familiar codebases were measurably slower when using AI assistance on complex tasks, despite predicting beforehand that the tools would speed them up. Afterward, even having just been slower, they still believed the tools had helped — the engineering equivalent of a driver flooring it right after the low-tire-pressure light comes on. Neither a test suite nor a green checkmark has any way to register that gap. A build doesn’t know whether the person or the agent that produced it was overconfident — it only knows whether the code ran.

Why the CI pipeline stays green while the codebase gets worse

Coverage percentage and a passing test suite tell you about one diff, in isolation, at one moment. They don’t tell you what’s happening to the codebase over time. One 2025 analysis of over 200 million lines of committed code found that duplicated code blocks rose roughly eightfold, and the share of code rewritten within two weeks of being committed roughly doubled, over the period AI coding assistants went from a novelty to a default. Over the same stretch, deliberate refactoring — the maintenance work that keeps a codebase navigable — fell from about a quarter of all changed lines to under a tenth. It’s the codebase equivalent of five people bringing the same casserole to Thanksgiving because nobody checked what was already on the table.

None of that trips a red X. Lint passes on duplicated code. Tests pass on a function that will be rewritten again in nine days. A CI pipeline evaluates each pull request as if it exists in isolation, so a codebase can get measurably harder to maintain while every individual commit that caused the decline sailed through review with a green light.

The bug class your linter can’t see

Independent testing published in 2026, run across five current AI models, found that somewhere between one in 22 and one in 16 code completions still referenced a software package that doesn’t exist — a plausible-sounding name the model invented rather than something pulled from a real library index. Dozens of the invented names were still unregistered, meaning anyone could claim one and wait for a model to keep recommending it to unsuspecting developers, a supply-chain risk now commonly called slop squatting — somewhere out there, a domain squatter owes several AI labs a thank-you card.

A hallucinated import is syntactically valid. It reads correctly to a linter and often survives a test suite that mocks its dependencies. It only fails at install time, or when someone registers the name a model keeps guessing — which means the tooling most teams already run isn’t positioned to catch it at all.

How to actually close the gap

The fix isn’t a smarter model or a stricter linter. It’s a small set of additional gates, each aimed at a specific failure mode above, and none of them require new tooling budget.

Replace coverage with a mutation score for changed code. Line coverage tells you code executed, not that a wrong result would be caught. A mutation-testing tool deliberately injects small, artificial defects into your code and checks whether your test suite notices. If it doesn’t notice a flipped comparison operator or a swapped plus sign, that’s not a test suite — that’s a formality with a green badge. Run it only against files changed in a given pull request — full-repo mutation runs are usually too slow for CI — and set a minimum mutation score (60–70% is a reasonable starting point) as a required check, the same way you’d require a passing test suite today.
Add property-based tests to your highest-risk logic. Example-based tests check one hardcoded input against one expected output. Property-based tests generate hundreds of random inputs and check that an invariant always holds — for a tag’s engineering-unit scaling function, for instance, asserting that the scaled value never falls outside the tag’s configured span, and that converting a raw reading to engineering units and back returns the original raw value within instrument tolerance, rather than asserting one example pair. Add these alongside your existing tests, not instead of them; recent research combining both approaches caught meaningfully more injected bugs than either method used alone.
Gate every new dependency behind a provenance check. Before a pull request merges, resolve every newly introduced import against your package ecosystem’s index and confirm it existed, with a real publish history, before the PR was opened. Anything that doesn’t resolve — or that was published within the last few days — gets flagged for a manual look rather than silently added to the lockfile.
Track duplication and churn as CI pipeline signals, not retrospective ones. Run a duplicate-code detector as a non-blocking report on every pull request, and flag files that get substantially rewritten more than once within two weeks of merging. Neither needs to block a build today; both are useful enough as visibility that most teams choose to act on the pattern once they can see it.
Require independent verification of anything an agent reports about itself. If an agent or a generated script reports “tests pass” or “migration complete,” have a separate process re-derive that specific claim — re-run the test, re-check the row count — rather than accepting the agent’s own summary. Treat “done” as a claim to verify, not a status update to trust — an agent reporting “migration complete” is the software equivalent of a teenager yelling that the dishes are done from the next room. Check the sink. This matters most for any agent with write access to a database, deployment target, or other mutating system; put a human-executed confirmation step between the agent’s recommendation and the action itself.
Add a calibration line to your review checklist. For AI-authored diffs specifically, ask reviewers to start from “assume this is wrong until the diff shows me why it’s right,” rather than the lighter skim a trusted colleague’s PR might get. Given that overconfidence shows up as the default response to AI-assisted code rather than an occasional failure, this one line does more than any additional automated check.

What to avoid

Don’t treat a coverage percentage as proof of correctness — it measures execution, not verification, and AI-generated tests are especially prone to asserting the wrong property while still going green.
Don’t give an AI-assisted PR a lighter review than a human-authored one just because it reads cleanly; clean-looking code that’s wrong is the well-dressed con artist of software defects — the suit was never the tell.
Don’t assume a newer or larger model has solved dependency hallucination — treat every new import as unverified regardless of which model produced it, since current testing still finds meaningful hallucination rates in the newest models available.
Don’t connect an agent directly to a mutating action — a deploy, a delete, a schema migration — without a human-executed step in between, and don’t accept the agent’s own account of what it did as a substitute for checking.

Final thought

The CI pipeline that missed our temperature-scaling bug wasn’t broken. It was answering last decade’s question perfectly well — it just wasn’t the question that mattered anymore. The fix wasn’t a smarter model or a longer test suite. It was asking a different question of the code sitting in front of us: not “did this run,” but “what would have to be true about this for me to actually trust it?” That’s a question worth asking about every green build on your screen right now — including the ones throwing confetti at you.

About the Author

I’m a Software Engineer at Cognizant with three years in quality engineering. My focus is agentic testing: AI tools that handle test generation, execution, and triage. I’ve shipped internally a PR reviewer and a test case generator, and I care about making agentic AI and QA accessible beyond traditional engineering paths. Connect with me on www.linkedin.com/in/dasari04.

Sources

Certainty Isn’t Correctness: The Real Cost of Trusting AI-Written Code

Be the first to comment

Leave a Reply Cancel reply