Software Testing Magazine

Why AI Models Pass Benchmarks but Fail in Production
Headline

Why AI Models Pass Benchmarks but Fail in Production

by Software Testing Magazine in Headline 0

There are many cases where AI models that worked fine in development start to behave badly in production. In this article, Kanika Vatsyayan discusses the causes that lead to this situation. He recommends moving from static assessments to continuous quality engineering for AI models. share share share share 

Featured

Software Testing Videos

Rethinking Software Quality: a New Approach to Testing API

October 9, 2026 0

This session, “Rethinking Software Quality: Why API Testing Needs to Change,” unpacks the challenges of testing API-driven ecosystems, the limitations of conventional QA, and the new pressures created by AI and evolving digital platforms.

Featured

Be nice: share this knowledge

Knowledge

  • ServiceNow’s Autonomous Engineer Puts AI in the Driver’s Seat — With Humans at the Guardrails

    October 14, 2026 0
    Paulius Pazdrazdys, a longtime observer of automated consumer-facing tech platforms, sees a clear pattern in how consequential automation tends to land in practice. Autonomous Engineer, ServiceNow’s newly announced offering, encodes that pattern directly: the system executes autonomous planning, building, and testing of implementation work, yet developers retain control of critical [...]

  • 7 Effective Ways Software Testers Can Retain Technical Knowledge and Testing Concepts

    October 13, 2026 0
    Software testing moves fast. One day you’re mastering a new automation framework. The next? You’re deep in the weeds of complex API status codes or security concepts, staring at a terminal screen late into the evening. share share share share 

  • Testing Duplicate Submissions in a Casino API

    October 12, 2026 0
    Duplicate delivery is normal in a distributed wallet system. A provider can send a wager, receive no response and send it again even though the first request has already reached the database. share share share share 

  • Why Pull Requests Should Review Tests, Not Just Code

    October 12, 2026 0
    When you add new code to your applications, it is associated to the test scripts that should validate it. Do you give enough importance to these tests? In this article, Eli Lopian explains that when pull requests are opened, you should review the tests scripts as much as the code. [...]

Tools

  • Turning Postman Collections into Committable Pytest Suites

    June 23, 2026 2
    In this article, Mikhail Golikov, the sole QA on a seven-team backend platform, explains how he turned a drawer full of unrun Postman collections into committable pytest suites, walks through the conversion request by request, and shows the folder-scoped command-line tool he built so one team’s pattern could scale to [...]

  • Open Source Test Data Generators

    June 15, 2026 10
    In most of your software testing activities, you need data. Sometimes you can rely on a small sample, but if you want to perform some load testing or if you want to test a feature that needs to produce a multipage invoice, then you start to need more than just [...]

  • Open Source API Testing Tools

    March 3, 2026 6
    The rising trend of using SOA and then microservices as a software architecture has led to the creation of multiple tools for automated testing of the services API. This article presents a list of open source API testing tools. share share share share 

  • Open Source Test Management Tools

    December 2, 2025 4
    Test management is defined by Wikipedia a part of the software testing process that includes the planning of tests and test cases, their execution and the storage and analysis of the tests results. This is achieved also by the integration with requirements management tools, functional software testing tools like Selenium [...]

Videos

Copyright © 2009-2026 Software Testing Magazine by Martinig & Associates