Swimlane · 09 of 10 · Remediation

Alerts that resolve themselves, or page with a tested hypothesis

The agent never acts on a hunch. It tests the hypothesis on a copy first, and when it cannot fix the problem the human still starts ahead: the page arrives with the hypothesis and the evidence.

Alerts that resolve themselves, or page with a tested hypothesis Swimlane across monitoring, a triage agent, the gateway and verifier, and an on-call human: an alert triggers read-only triage and a sandbox-tested hypothesis, a matching runbook executes a scoped fix that is verified and auto-closed, and any miss or failed verification pages a human with the evidence attached. MONITORINGTRIAGE AGENTGATEWAY / VERIFYON-CALL HUMANMATCHPASSNO MATCHFAILEDAlert firesSLO burn · page ruleAuto-closealert resolvedRead-only triageT0 · logs · metricsTest hypothesisreplay in sandbox copyRunbook + policytier decides who actsExecute fixscoped token · auditedVerify contractdid the SLO recover?Page a humanhypothesis + evidenceLEGENDHypothesis tested firstEscalationSuccess handoffFail path

Requirement

Incidents where the first responder is an agent arriving with a hypothesis already tested, and alerts that resolve themselves.

Design choices

  • Triage is read-only (T0) and cheap, so it can run on every page.
  • Fixes run only through pre-approved runbooks and the gateway; a failed verify escalates, it does not retry blindly.
  • Auto-close requires the verify contract, not the absence of an alert.

Evidence and gap

Built in Hangar: HolmesGPT triage and Sloth-based SLOs. Proposal only: sandboxed hypothesis replay and runbook execution through the gateway.

Hangar · reference architecture