Measuring a Counterpart Who Doesn't Understand "Pass/Fail" — Tricentis Ships a Prototype That Scores AI Agents by Probability and Returns a "Ship-or-Not" Verdict

As coding agents begin to take over implementation, the bottleneck in development is shifting from "writing" to "verifying." The faster code gets written, the harder it becomes for testing and review to keep pace. Tackling this head-on is Tricentis, a major player in test automation. On August 20, at its annual conference "Tricentis Transform" held in Dallas, the company unveiled three prototype features that turn quality assurance (QA) itself into agents. All are offered through the early-access experimental program "Tricentis Labs," and the company adds the caveat that these are "features under development and not a commitment to future commercial availability."

Why Push Even the "Testing Side" Toward AI?

Traditional testing was a "deterministic" task: a human wrote the test cases, defined the expected values, and judged pass or fail. It rests on the premise that the same input yields the same result. But the code an agent generates, and the behavior of the agent itself, won't necessarily produce exactly the same result every time. Against this "non-deterministic" counterpart, the old pass/fail judgment no longer fits as-is. Tricentis's answer was to place agents within the testing process itself and to rebuild how judgments are made.

The Three Prototype Features Unveiled

"Aida": Roaming an App Without Scripts

Aida is an agent that autonomously explores web applications and Windows desktop applications. Its defining trait is that it requires no pre-built test scripts or initial setup. Even in areas where no human-written test assets exist, it operates the app on its own, roams around it, surfaces defects and testing gaps (coverage gaps), and assesses the app's health. Its intended use cases are screens thinly covered by existing tests and exploratory testing in the early stages of new development.

"AgentScore": Grading Agents by "Probability"

This is the newest idea of the bunch. AgentScore observes AI agents within actual workflows and evaluates their behavior. Rather than a deterministic pass/fail, it shifts its footing to probabilistic evaluation: it computes a composite quality score and then returns one of three recommendations — "review," "block," or "ship." Including suggestions on what should be measured, it provides the material for deciding whether a non-deterministic agent is "okay to release." As more agents write code, it stands as one answer to the question of who vets that code, and how, before it goes out into the world.

"Release Risk Intelligence": Surfacing Gaps at the Release Level

This feature is aimed at release managers and heads of quality engineering. Scoping to a specific release, it makes coverage gaps visible, ranks risks by severity, and has AI advise on the next actions to take. It supports prioritizing where to focus verification within limited time.

A Move to Grasp Context — the Tabnine Acquisition

Behind all this lies the late-July acquisition of Tabnine, a pioneer of AI coding assistance. The aim is to embed Tabnine's "Enterprise Context Engine" into the testing agents. This technology extracts entities, dependencies, and architectural patterns from code repositories, documentation, tickets, APIs, and infrastructure metadata, and continuously updates a knowledge graph specific to that company. By giving the testing agents "the company's own context," the goal is to make them behave more accurately than a general-purpose AI. Tricentis cites customer-reported figures where introducing this context engine improved AI accuracy by up to 2x, cut token consumption by up to 80%, and resolved complex tasks up to 50% faster (note that these are figures for the context engine alone, not the performance of the new features announced here).

The Impact on Development Teams and Business

The more "writing agents" like Claude Code and Codex spread, the more testing and review throughput becomes the relative constraint. If the generation, execution, and risk assessment of tests can all be shifted onto agents, a picture emerges of running the entire software development lifecycle (SDLC) through agents. As a realistic entry point into production, it makes sense to start with early access via Labs, limiting trials to exploratory testing in areas without existing test assets and to ship-or-not judgments for agents run in-house.

Guardrails Against Overconfidence Are Needed Too

At the same time, probabilistic grading is not a "guarantee of passing." Review, block, and ship are only recommendations; the final shipping decision, and the responsibility for it, remain with humans. All of these features are at the "under development, commercialization uncommitted" stage, and rushing them into production is ill-advised. There is also the question of who ensures — and how — the validity of an agent that evaluates other agents: the problem of who watches the watchmen. Without transparency in the evaluation criteria and ease of auditing, one can point to the risk that overreliance on automated testing actually lets human oversight slip away. Beneath the convenience, what's called for is a stance of adopting these tools incrementally while keeping an eye on such concerns.

References: Tricentis Official Blog / Yahoo Finance / VKTR / DEVOPSdigest / Tricentis (Tabnine acquisition)