Process · 10 of 10 · Evaluating infrastructure agents

Evaluating infrastructure agents: a regression suite, not a vibe

Every model, prompt or tool change is replayed against the same corpus in disposable sandboxes and scored by the same verifiers that gate production. If it is worse than the baseline, it does not ship.

Evaluating infrastructure agents: a regression suite, not a vibe Process diagram of an evaluation loop: a versioned corpus of replayable incidents and tasks runs an agent under test in a fresh sandbox, a deterministic scorer records outcome, blast radius, policy violations and cost, a regression gate ships or blocks the model, prompt or tool change, and blocked changes loop back for revision. REPLAYRUNTRACESCORERECORDVERDICTNEW VERSIONBLOCKED: REVISE AND RESUBMITTask corpusreplayable · versionedDisposable sandboxfresh per runAgent under testmodel · prompt · toolsVerifier scoreroutcome · blast radiusRegression gateno worse than baseScorecard historyper versionShip or blockthe changeProposed changemodel · prompt · toolSame corpus for every change,scored by verifiers, not by another model.LEGENDJudge of recordThing under changeHistoryChange enters loopFeedback

Problem

Infrastructure agents need an evaluation that is as rigorous as the gates their changes have to pass.

Design choices

  • Corpus: past incidents and golden-path tasks, replayable against a fresh sandbox.
  • Scored on outcome, blast radius, policy violations, retries and cost, all by deterministic verifiers.
  • Scorecard history makes model and prompt regressions visible per version.

Evidence and gap

Built in Hangar: the verifier side (gate checks, verify contracts, release records). Proposal only: the replay corpus and harness. This is the design to build first; none of it is built.

Hangar · reference architecture