Your Agent Eval Should Survive a Vendor Swap

The agent pilot works. Then the model price changes, a framework loses support, or another team standardizes on a different stack. Engineering can move the code. Product asks a more uncomfortable question: will the replacement have to pass the same test?

Too often, the answer lives inside the old vendor's dashboard. The prompts are stored there. The graders expect its trace format. The failure history cannot be replayed without rebuilding the pipeline. What looked like product evidence turns out to be an integration.

An agent is not production-ready until its core evaluation can travel. The same task, outcome, forbidden actions, case set, trace evidence, and release threshold should run against a replacement model or framework without changing what success means. This is testable. Run the same suite before a swap and measure coverage loss, migration time, critical-failure detection, and decision consistency.

The stack will change before the user job does

AWS made this problem unusually visible on August 26. Its new AgentCore Evaluations integration can score agents built with several frameworks when their traces follow OpenTelemetry or OpenInference conventions. The important product signal is not the AWS service itself. It is the separation of the evaluation contract from the framework that produced the run.

The market is already giving teams reasons to preserve that separation. OpenAI's AgentKit announcement now says its Agent Builder and Evals products will be unavailable after November 30, 2026, with different successor paths depending on the workflow. That does not make the original product a mistake. It does show that a vendor surface can have a shorter life than the product decision it helped a team make.

The user's job is usually more durable. A support agent still needs to resolve the right issue. A research agent still needs to ground its answer. A workflow agent still needs to use the authorized tool and leave the system in a known state. Those outcomes should not be renamed every time the orchestration layer changes.

Portable does not mean generic

Portability fails when teams reduce an eval to one vague score that every system can emit. "Helpfulness" can travel easily and still miss the failure that matters.

The portable unit should be a product contract with observable parts:

• the task and terminal state;
• the allowed fallback and forbidden action;
• representative, edge, adversarial, and regression cases;
• the tool calls and state changes needed to reconstruct the run;
• the grader mix and protected slices;
• the threshold that ships, holds, or rolls back the release.

OpenTelemetry's semantic conventions provide shared names for agent operations, messages, tool calls, results, model versions, and usage. They are plumbing, not a product metric. A common trace can tell two systems to expose comparable evidence. The team still has to decide which outcome is correct and which failure blocks release.

That distinction matters. Standardize the evidence envelope. Keep the judgment specific to the job.

The Portable Agent Eval Contract

The following artifact is a proposed starting point for one consequential agent task. Each field should remain readable outside the vendor console.

The last field keeps the contract honest. A document is not portable because it says so. It becomes portable when another stack can run it and produce evidence that supports the same decision.

A synthetic support agent exposes the gaps

Consider a synthetic employee-support agent that handles account-lockout requests. The task is complete only when identity checks pass, the authorized unlock tool succeeds, the employee can sign in, and the ticket records the outcome. Asking for human help is an allowed fallback. Changing another user's account is a critical failure.

The first implementation records only the final response and a vendor grader's helpfulness score. It passes. During a framework swap, the team discovers that it never preserved the identity-check result, tool arguments, account target, or final ticket state. The new agent can produce an equally reassuring answer while acting on the wrong account. The old score cannot distinguish the runs.

Apply the contract and the evaluation changes. The case fixture travels with the product. Deterministic checks compare the requested and affected identities. The trace records the tool and result. A task grader checks the completed state. Human review examines ambiguous recovery cases. Both implementations face the same critical-failure rule.

The goal is not identical prose or identical traces. The goal is equivalent evidence about the user outcome and the dangerous paths.

The common layer can hide useful detail

There is a serious counterargument. A portable core can become a lowest-common-denominator suite. One framework may expose planner revisions, retrieval choices, memory writes, or guardrail events that another cannot. Forcing every signal into a shared shape can discard the best diagnostic evidence. Building adapters and rehearsing swaps also costs time that a small team could spend fixing the product.

The answer is a two-layer design. Keep the release-critical contract portable, then add stack-specific diagnostic checks around it. The portable layer answers whether the product still meets its promise. The local layer helps explain why a particular implementation failed.

NIST's draft TEVV-Athlon framework supports this kind of tailoring. It proposes a flexible method for building assessments around organizational objectives across different AI systems. The draft does not prescribe this contract, and it is still open for public comment. Its useful boundary is that evaluation should fit the objective rather than force every product into one universal test.

The thesis weakens if vendor-specific suites repeatedly migrate with less work, no loss of critical coverage, and better failure detection than a portable core. Measure that instead of treating portability as an article of faith.

Rehearse the replacement before you need it

Choose one high-consequence task and one credible alternate stack. Export the cases, replay them, and mark every field the replacement cannot reproduce. Then decide whether the gap is a product risk, a diagnostic loss, or harmless implementation detail.

Do this while the current system still works. A rushed migration is the worst time to discover that the team's definition of quality belongs to the tool being replaced.

The model and framework are implementation choices. The product promise is not. For the companion release gate that defines how an eval becomes a shipping decision, read The AI Feature Ship Gate.