Two years ago, a regression cycle on one of our client releases ate most of a Friday. The nightly suite came back red in dozens of places, and our QA team spent the morning sorting real defects from broken locators and a staging database that timed out on a whim. Flaky tests got rerun twice. Exploratory testing got whatever was left of Monday. On a recent release, the same cycle arrived pre-sorted: failures bucketed, locators patched and flagged, visual noise filtered out. AI did not shrink QA. It moved QA from executing checks to judging them.
That sounds like a small change. It isn’t. The release went out faster, but not because fewer people touched it. The same engineers spent their hours on different questions. Is this auto-classified “automation bug” really an automation bug? Did the healed selector land on the right button? Is the visual diff a regression or a design change nobody announced? I call this the verdict shift: the machine runs, sorts, and repairs, and the QA engineer rules on whether what it did is true.
Most of the debate about AI in testing still argues over the first half of that sentence. The interesting part is the second half.
The tooling took over the repetitive half of the job first
Test generation went from conference demo to shipped feature fast. In October 2025, Playwright 1.56 introduced Test Agents: a planner that explores the app and writes a Markdown test plan, a generator that turns the plan into test files while checking selectors live, and a healer that runs the suite and repairs failing tests. Read the healer docs closely. If it decides the functionality itself is broken, it can skip the test instead of fixing it. That’s a model handing down a verdict on whether your product has a bug.
Locator maintenance went the same way, only earlier. Healenium wraps a Selenium driver in a SelfHealingDriver, catches NoSuchElement failures at runtime, and uses a machine learning model to pick the element that best matches the old one. Its IntelliJ plugin can write the healed locator back into the codebase. Useful. Also a very quiet way to make a test pass against the wrong control.
Failure triage is where we felt the change most. ReportPortal’s Auto-Analysis takes failures still marked To Investigate, pulls similar historical logs that people already classified, and lets an XGBoost model assign the defect type when its probability reaches 50%. The item gets an AA label and a comment copied from the closest past match. A 50% call is a confident coin flip. It saves hours anyway, as long as someone samples what it decided.
Visual comparison stopped being pixel math. Applitools Eyes offers match levels: Strict catches changes in text, font, color and element position while ignoring platform rendering noise, and Layout only checks whether elements appeared, disappeared or moved. Picking the level per screen is test design. The AI does the comparing. A person still decides what counts as a difference.
We see this from both sides. The LITSLINK teams that deliver custom AI development services build classifiers, recommendation engines and LLM features for clients, and our QA team tests those same products on the same sprint board. So the tools above are only half the story for us. The other half is what happens when the thing under test is itself a model.
Testing an AI feature is a different job, not a harder version of the old one
Deterministic code gave us a gift we stopped noticing: the same input produced the same output, so an assertion meant something. LLM features take that gift away, and not only because of sampling temperature.
In September 2025, Horace He and colleagues at Thinking Machines ran the prompt “Tell me about Richard Feynman” through Qwen3-235B 1,000 times at temperature 0 and got 80 unique completions . The cause wasn’t randomness in the model. Server load changes batch size, and inference kernels that aren’t batch-invariant return slightly different numbers depending on what else is in the batch. Temperature 0 is not a determinism switch. A snapshot assertion on raw model output is a flaky test you wrote on purpose.
So the oracle changes shape. Instead of one expected value, you need an evaluation set: a few hundred real inputs, a scoring rule for each, and a threshold agreed before the run. Some scoring is cheap (did the JSON validate, did the answer cite the retrieved document). Some is expensive, like a rubric graded by a person or by a second model that itself needs spot checks. Here’s the problem. Most regression suites have no place to store “this case passed with a score of 0.82 last week and 0.74 today.”
Hallucination makes it worse, because the failure looks like success. The response is fluent, formatted and confidently wrong. Tests for a retrieval feature have to check grounding: does every claim trace to a source passage, and does the system refuse when the context doesn’t contain the answer?
Then there’s drift, which breaks the oldest assumption in regression testing, that nothing changes unless someone commits. Lingjiao Chen, Matei Zaharia and James Zou compared two versions of GPT-4 in 2023 and found its accuracy on telling prime from composite numbers fell from 84% in March to 51% in June. Same product name. Different behavior. If your feature calls a hosted model, your dependency can regress while your repository sits untouched.
Bias is the last piece, and aggregate scores hide it well. A speech feature that scores fine overall can fail on one accent. A ranking model can treat two user segments differently with no single test turning red. You only see it when you slice the evaluation results by group, and someone has to decide which groups matter.
Yevhenii Melikov, Machine Learning Engineer at LITSLINK, put it this way when we planned evaluation work on a recent model release:
“With normal code I write an assertion and it holds or it doesn’t. With an LLM feature I need an evaluation set, a scoring rule, and a threshold we agreed on before the run, because the same prompt can come back worded differently tomorrow. What I want from QA is ownership of that set: which cases go in, who labels them, and how big a score drop has to be before it blocks a release.”
Notice what he didn’t ask for. He didn’t ask QA to write prompts or tune models. He asked QA to own the definition of correct, which is what QA has always owned, just with a wider target.
The “AI will replace testers” line gets the direction backward
“Give it two years and AI will write, run and fix every test. Manual QA is finished.”
That’s the line you hear at every conference with an AI track, usually right before a demo of an agent generating forty tests from a login page. The demo is real. The conclusion isn’t.
Look at what developers report about the tools already in their hands. The Stack Overflow 2025 Developer Survey found that 84% of respondents use or plan to use AI tools, up from 76% the year before. In the same survey, 46% said they distrust the accuracy of AI output, against 33% who trust it, and 66% named “AI solutions that are almost right, but not quite” as their top frustration. Another 45.2% said debugging AI-generated code takes more time.
“Almost right” is exactly what a test exists to catch. When code generation speeds up, the volume of plausible, slightly wrong output goes up with it. When test generation speeds up, you get more tests whose expected values someone has to confirm. Neither trend removes the need for judgment. Both multiply it.
Not fewer testers. Not cheaper testers. Testers doing a different job.
The replacement argument also confuses two things that look alike on a dashboard. Executing a check is a commodity, and has been since the first CI server ran a suite at 2 a.m. Deciding whether a check is the right check, whether its oracle is sound and whether its failure matters is not a commodity. AI took the first one. It made the second one larger.
Here’s where each piece of the work actually went
This is the split we now plan sprints around. The middle column is what the tooling handles on our projects. The right column is what nobody should hand off.
| QA task | What AI now does | What the QA engineer now owns
|
| Regression test authoring | Drafts test plans and test files from app exploration or requirements | Reviewing assertions, rejecting tests that check the wrong thing, coverage decisions |
| Locator maintenance | Heals broken selectors at runtime and proposes code patches | Approving each heal, spotting heals that pass against the wrong element |
| Failure triage | Classifies failures by matching them to past investigated logs | Sampling classifier output, correcting labels, tracking where it guesses wrong |
| Visual regression | Compares screens with perceptual matching, ignores rendering noise | Choosing match levels per screen, deciding which diffs are intended |
| Flaky test handling | Flags unstable tests from rerun history | Root-causing the flake and deciding quarantine versus fix |
| Exploratory testing | Suggests risky areas from change history | Running sessions, following hunches, writing up what the scripts missed |
| AI feature testing | Scores outputs with automated metrics or model graders | Building evaluation sets, setting thresholds, slicing for bias, monitoring drift |
Read the right column top to bottom and it describes a reviewer, an analyst and a product skeptic. It does not describe someone clicking through a test case in a spreadsheet.
Adoption is wide, maturity is thin, and that gap is the QA job
The best current read on how far this has gone comes from the World Quality Report 2025, published by OpenText, Capgemini and Sogeti in November 2025 from a survey of more than 2,000 senior executives in
22 countries and 10 sectors.
- 89% of organizations are piloting or deploying GenAI-augmented workflows in quality engineering. 37% are in production and 52% are still in pilots.
- Only
15% have reached enterprise-wide implementation, and 43% remain experimental. - The average reported productivity boost is 19%, and about a third of organizations have seen minimal gains.
- 60% cite hallucination and reliability concerns as a top challenge, behind data privacy (67%) and integration complexity (64%).
- 50% say their organizations lack AI/ML expertise, the same share as in 2024.
- The non-adopter rate rose to 11% in 2025, up from 4% in 2024.
Put those numbers next to each other and a pattern shows up. Almost everyone has started. Few have scaled. Gains are real on average and absent for a third of teams. Reliability is a top worry, and the skills gap didn’t move in a year. My read is that the 19% lands where someone owns the verdicts and stalls where teams expect the tool to own them. That last part is my inference, not the report’s. It matches what we’ve seen on client projects, where the AI tooling paid off fastest on teams that treated its output as a pull request, not a result.
A checklist for QA leads for the next two quarters
- Every AI-made change to a test goes through review. Healed locators, generated tests and auto-skipped cases should land as diffs a person approves, the same way
production code does. A self-healing suite with no review step is a suite that slowly stops testing anything. - Measure your triage classifier like the model it is.
Pull a weekly sample of auto-labeled failures and track how often the label was right, per defect type. If “system issue” is wrong a third of the time, you’ll know where to stop trusting it. - Ship no AI feature without an evaluation set QA owns.
Start with real production inputs, add the edge cases support tickets keep surfacing, and agree on the pass threshold with product before the first run, not after a bad one. - Retire exact-match assertions on model output.
Replace them with schema checks, grounding checks and scored rubrics. Keep the snapshot tests for deterministic code, where they still earn their keep. - Run evaluations on a schedule, not only on commit.
Hosted models change under you. A weekly eval job against a pinned set catches drift before a customer files the ticket. - Spend the saved triage hours on exploratory and risk-based testing.
That time was the point of the tooling. If it quietly turns into more ticket grooming, you bought automation and kept the old bottleneck. - Train the team to read model behavior.
Prompt writing is the easy part. Interpreting score distributions, spotting slice-level failures and arguing about thresholds with an ML engineer is the skill half of the industry still says it lacks.
The checks got cheap, and the verdicts got expensive.
Leave a Reply