Docs · Hangar
Hangar roadmap: Airframe A+, Autopilot, and the Skyport AI workloads
This is the unified plan. It replaces the “next” section of the 2026-09-25 Skyport handoff and folds in three streams that used to be separate: Skyport’s remaining phases, the Airframe A+ program, and Autopilot. All durations are estimates for one person; ordering matters more than the numbers.
Diagrams: diagrams/plan/01-hangar-family.html, 02-roadmap.html, 03-dependencies.html,
04-scorecard.html.
1. The shape of the plan
Skyport (demo + quickstarts) ──────────────┐ builds new components → born A+
Airframe A+ (AF-1 … AF-10) ─────────────────┼── contract first, so nothing is retrofitted
Autopilot (Clearance, AgentRun, tools) ────┤ depends on the contract and ownership split
Skyport AI workloads (parts 6-11) ─────────┘ depends on Autopilot phases A and B
Since 2026-10-28, Clearance runs its own product track beside this plan (the Clearance Roadmap: discovery in Nov 2026, the deployment gate on hangar-prod in Dec 2026, recorder, ledger and packet in Jan 2027, partner pilots, go/stop at the end of Apr 2027). This plan builds Clearance’s runtime gateway, which Autopilot needs either way. The deployment gate is that track’s work, and the Hangar lab is its reference deployment. design.md section 0.1 records what it changes here.
Why this order. Three reasons, each of which would cost weeks if ignored:
- Build the contract before the next components. MongoDB and OAuth are next in Skyport. If they ship before the contract exists, each needs a retrofit for outputs, verify checks and schema strictness. Ship the contract foundation first and they are born A+.
- Ownership before agents. An agent that can edit a file mixing human and machine-owned keys is a risk that path-level scope cannot contain. Split the files (AF-5) before granting Clearance write tools.
- The planner is the acceptance test. The parachute sentence (the one-line request to an agent in airframe-ai-friendly.md: a new Python app named parachute, with dev, test, staging and prod environments, a canary step and one env var) needs AF-1 to AF-6 and the Autopilot core. Everything before it is a prerequisite; everything after it (Skyport AI) is a consumer.
What changed from the 2026-09-25 plan:
| Old | New |
|---|---|
Phase 2 remainder: baggage-api next |
Still first (M0), and used to dogfood airframe validate |
| Phase 3: MongoDB | M1, built as the first born-A+ component |
| Phase 4: OAuth | M2, born A+ |
| Phase 5: nginx (optional) | M5, born A+ |
| (none) | Airframe A+ program, Autopilot, six AI workloads |
2. Milestones
Score targets are the scorecard’s overall (baseline 27/100, A+ needs 97 and 14 of 14 checks).
M0 Stabilize and baseline (weeks 1-2), scorecard target 35
| Task | Notes |
|---|---|
| AF-10a chart guard: no Rollout until an image exists | worktree, test-via-copy on a scratch app; then flip Features.rollout_guard; the canary test in tests/test_airframe_plan.py fails on purpose when it lands |
| AF-10b chart tests and chart CI; scorecard in CI | baseline is committed; CI fails on regression |
AF-1a AGENTS.md at the Airframe root |
draft in docs/autopilot/drafts/airframe/AGENTS.md |
AF-4a airframe validate v0: strict-derived schema and helm template |
use it on baggage-api’s env files as they are written |
SP-2r baggage-api (Python) through the current flow |
record every stumble as a scorecard item; this is the manual version of the parachute sentence |
| AP-0 answer the unverified list (section 5) | 30 to 60 minutes each; they de-risk M2 and M3 |
DX-0 decide and create the autopilot repo |
move ~/tech/autopilot there; pytest as CI |
Exit: the guard is live on dev; airframe validate catches an injected typo on a real PR; baggage-api runs on dev; every item this milestone’s own packet actually asks for (U1, U5, U6, U7, U11 - see task 1 above) has an answer. (U2, U3, U8, U9, U10 are follow-on experiments that de-risk M2/M3/M5, not M0 gates - the packet was written that way; this line originally over-promised “every item in section 5” and is corrected to match.)
M0 status (2026-09-27): 100% done against its own exit bar; scorecard 27.3 -> 41.3 (target 35).
- Done: AF-10a guard (Airframe v0.3.91, pinned on the dev and prod clusters; all 7 live env files render byte-identically before and after); AF-10b chart tests and CI (the scorecard job needs a
FLEET_READ_TOKENsecret); AF-1aAGENTS.md; AF-4aairframe validatev0; DX-0autopilotrepo (public, github.com/jfillman/autopilot); planner flipped (Features.rollout_guarddefaults to True, the canary test retired); D7 decided (Preflight); D6 and D10 briefed (briefings-d6-d10.md); D10 rollout itself now live too (thevaluesrelease-guardrail is mandatory on all 3 candidate gitops repos, not just briefed); release-file split designed and proven on the dev cluster (release-file-split.md), and its D9/AF-5 continuation (schema field, both writer Tasks, the 3-valueFilesApplicationSet change) shipped and live-verified on both clusters 2026-09-27 (see that doc’s own Progress table - this runs past M0, not gated by it). - Exit criterion “
airframe validatecatches an injected typo on a real PR” demonstrated live 2026-09-27: a fixture PR againstgitops-baggage-api(m0-typo-proof, closed after) withrolout:injected intoprod/staging/values.yamlmade thevaluesrelease-guardrail Check fail for real, not just offline. - U1, U5, U6, U7, U11, U12 answered and U3 partly (section 5); U4 also now answered (2026-09-27: Tower’s Re-run/Cancel confirmed against a real failed PipelineRun on
baggage-api). Still open, not required for M0: U2 (function TTL re-invoke), U8 (needs theAgentRuncomposition applied via a copy), U9/U10 (deferred by the user’s own choice, M2/M5). baggage-api: fully working end to end, dev and prod staging bothSynced/Healthy, broker-connected, image0.1.0-ae82839(Python versioning fix live). The first two image scans failed (Debian base: 44 unfixed HIGH CVEs; then a stale setuptools), which is a scorecard item for the Python scaffold.
M1 Contract (weeks 3-6), target 60
| Task | Notes |
|---|---|
| AF-2 strict schema, generated from one source | warn first, sweep the fleet, then enforce; descriptions to 95%; discriminated components[] |
AF-3 component outputs and fromComponent |
Redis, Postgres, RabbitMQ, SecretStore |
AF-4b airframe validate layers 3 to 6, dead-end rules seeded |
required check on app, gitops and tenants repos |
AF-1b contract bundle, llms.txt, XRD summaries |
the generator from AF-2 emits the bundle |
| AF-6a status helper and reason codes | on the XRDs touched in this milestone |
SP-3 MongoDB component, then baggage-api on it (part 4 complete) |
first born-A+ component: outputs, verify, sidecar meta, AGENTS.md, every field described |
Exit: typo acceptance is 0% across every live file; the Mongo component ships with its contract; validate is a required, green check on three repos.
M1 status (2026-09-27): AF-2/AF-3/AF-4b/AF-1b (#6-#10) all merged, tagged v0.3.95, live-verified on
both clusters and on a real PipelineRun. Airframe v0.3.95 tagged; glidepath-catalog’s
valuesValidatorImage/chartVersion bumped (pushed direct to glidepath main, its own convention).
gitops-cluster-dev’s idp-service-catalog Application had a real live-cluster risk found and fixed
before the pin moved: its xrds/*.yaml include glob was a wildcard that would have caught AF-3’s new
xrds/*.meta.yaml sidecar files as invalid manifests - narrowed to an explicit 13-file list (verified
by simulating the glob against the real tag: exactly 13 XRDs + 13 compositions, zero .meta.yaml).
gitops-cluster-prod’s equivalent already used an explicit list, so only needed the bump. Both
synced Synced/Healthy with the expected resource counts; the live Redis XRD carries its new
hangar.io/agent-summary. A real fixture PR against gitops-baggage-api (closed after) confirmed the
values guardrail catches AF-SECRET-001 (a GitHub-token-shaped literal) and warns AF-COMP-003 on
baggage-api’s own real unmigrated component references, on a live PipelineRun with a fail-reporting
GitHub Check - both brand new in this batch, not a repeat of the M0 typo check.
- AF-2, merged.
values.schema.jsonis the single hand-authored source (additionalProperties:false baked in everywhere except the documented passthroughs; descriptions 5% -> 100%);values.yamlis generated from it, CI-gated.tools/airframe-validateruns one pass against a compiled schema (tools/airframe_schema.py) with a discriminatedcomponents[].typeunion spliced in from the real XRDs. Found and fixed a real latent gap:cronJobs[]/jobs[]were missing several fields the strict form would have wrongly rejected. 48/48 injected mutations rejected on the real fleet. A follow-on fix (jfillman/airframe#7, also open) bakes the newairframe_schema.pymodule into the validate image - #6 alone would have shipped a broken image. - AF-3 (
jfillman/airframe#8), component outputs andfromComponent. Redis/PostgreSQL/RabbitMQ each declare their real outputs inxrds/<type>.meta.yaml, verified against their actual Compositions.env: [{name, fromComponent: {name, output}}]resolves to a real valueFrom, withAF-COMP-002lint rejecting a bad reference. A real migration PR againstboarding-api(branchaf3-fromcomponent-migration, staged, needs a user-signed commit - gitsign/Sigstore needs an interactive browser login this session couldn’t complete) replaces its hand-writtencache-master/board-mq-connection/board-mq-user-credentialsliterals, dropping a manually-Infisical-synced Redis password in the process. - AF-4b (
jfillman/airframe#9), partial. Five dead-end rules (AF-CLUSTER-001,AF-ENV-001,AF-COMP-001,AF-SECRET-001, andAF-COMP-003- an advisory that fires exactly on the fleet’s remaining hand-written component references) plus--format json. Not done: L3 kubeconform conformance, SARIF output, AF-ENV-002/AF-PROBE-001/AF-ARCH-001/AF-RABBIT-001 (need cluster/build context this session didn’t reach), and widening the required check from gitops-only to app + tenants repos (Tekton/CI wiring, not anairframe-validatechange). - AF-1b (
jfillman/airframe#10), done, stacked on #9.hangar.io/agent-summaryon all 13 XRDs;contract/airframe-contract.json(generated from the same sources the chart/validator use);tools/airframe-capabilities(a CLI answering component/XRD/field/output questions from the bundle alone);llms.txt.tools/test_capabilities_coldstart.pyproves 10 real capability questions answered correctly from the bundle, cross-checked against source. Not done: theadd-to-catalogBackstage-template review (a real product decision, left for the user) and wiring the 10-question proof into the separateautopilotrepo’s own Preflight harness. - AF-6a (
jfillman/airframe#11), done, stacked on #9. Redis/PostgreSQL/RabbitMQ each set a customComponentReadycondition (deliberately not namedReady- that’s Crossplane’s own core-managed type) with a closed reason set, computed from the composed child resource’s own real status. Live copy-composition-verified for every branch on all 3 components, both RabbitMQ modes - including a real transientPostgreSQLDegradedstate caught mid-bootstrap, not simulated. - SP-3 (
jfillman/airframe#12), the MongoDB component built and live-verified, born A+ - stacked on #11. DEDICATED mode, wrapping MongoDB Controllers for Kubernetes’MongoDBCommunityCRD (the design’s originally-named operator is archived; its unified successor preserves the same CRD, verified against the real repo, not memory). Outputs, verify checks, sidecar meta and aComponentReadycondition all shipped in the same PR, not retrofitted. Getting one real replica set running on dev surfaced three non-obvious cluster-infra gaps ingitops-cluster-dev, each fixed with the user’s explicit confirmation before the more sensitive RBAC grants: the operator neededwatchNamespace: "*"(app namespaces, not its own), its database pods needed two ServiceAccounts + a narrow Role/RoleBinding replicated per app namespace (the chart only creates them in its own namespace), and Crossplane itself needed two new RBAC grants for that - the second (pods: get/patch/deletecluster-wide) required by Kubernetes’ own RBAC-escalation check before Crossplane could delegate that permission via the Role it creates. End-to-end proof: a real replica set reached phaseRunning, the pod went2/2 Running, and the output Secret’s credentials were read and confirmed correct, not just checked for existence. Not done: migratingbaggage-api’s own application code onto it (real Node.js changes in a separate repo, replacing its current single-replica in-memory state) - the component itself is the whole scope of this PR; the app migration is real follow-on work, not started. - XRD-level descriptions/CEL rules are still at the M0 baseline in most XRDs (AF-1b’s own summaries don’t count as the per-field description sweep AF-2 originally scoped there).
boarding-api’sfromComponentmigration merged (jfillman/boarding-api#9). Merging it exposed a real gap: the chart’slower-envs/tenant-onboarding/tenant-identityApplicationSet pins - a third, easy-to-miss pin location forairframe-applicationbeyond Glidepath’schartVersionand each cluster’sidp-service-catalogsync pin - were still atv0.3.94(dev)/v0.3.92(prod), predatingfromComponent. The liveboarding-api-devpod crash-looped (REDIS_URL="<nil>") within seconds of the merge syncing; caught immediately by watching the Rollout, fixed within minutes (gitops-cluster-devf5d15b3,gitops-cluster-prodda0f5cfpreventively), confirmed by the actual pod going healthy and its logs showing it consuming events normally. See [[feedback_airframe_chart_three_pin_locations]] (memory) and the handoff’s live-incident block for the full timeline - this is now a standing checklist item for the next chart-behavior release.- Immediate next steps: merge
#11and#12(each stacked on the last); decide onadd-to-catalogfor the 7 newly-annotated XRDs (a real product decision, not blocking); migratebaggage-apionto the new MongoDB component (real app-code work, not started); M1’s remaining item after that is just the XRD-level description/CEL sweep.
M2 Safe write (weeks 7-10), target 75
| Task | Notes |
|---|---|
| AF-5 base layer and release-file split | migrate dev first, then prod, each via a scratch app; coordinate airframe, glidepath’s open-release-pr and deploy stage, and Tower’s Config tab |
| AF-5b ownership gate, risk classes, comment-preserving patch engine | |
AF-9a field-level agent-scope specified and tested |
Clearance policy grows a JSON-pointer allow and deny |
SP-4 OAuth component, skyport-auth, enforced JWTs (part 5) |
born A+ |
| AF-7a walkthrough runner; convert quickstart parts 1 to 3 | each step: command, expected result, verify |
Exit: no live file mixes owners; a release PR and a config PR never conflict (property test); part 1 replays green from a walkthrough.
M2 status (2026-09-29): all five workstreams done. Exit bar met - M2 is closed.
- AF-5, done for the 3 pre-existing live apps (
baggage-api,boarding-api,flight-api): dev-env split merged (baggage-api#4,boarding-api#10,flight-api#10); staging split (gitops-baggage-api#16,gitops-boarding-api#17,gitops-flight-api#10) built directly rather than waiting for each app’s next real release, since the ApplicationSet mechanism was already live.skyport-brokerhas no image to split (rollout: null), so it’s out of scope.platform/→airframe/(step 4, the directory rename) is deliberately not part of this - scoped separately, see below. - AF-5b, the doc-backed half done, two PRs:
airframe#15promotesAF-OWNER-001from warning to error now that the split is real fleet-wide (a release key in a human file, or anything else in a release file, both fail);autopilot#3adds the matching Clearance-side guarantee -release.yaml/*.release.yamljoinBASELINE_DENY_PATHS, un-removable by any agent definition. The roadmap row’s other two items (“risk classes”, “comment-preserving patch engine”) have no design anywhere in this repo - not attempted rather than improvised. - AF-9a, done,
autopilot#3(same PR as AF-5b’s Clearance half).clearance/scope.pygaineddiff_pointers(RFC-6901 JSON pointers for whatever changed between two parsed file contents),field_violations/field_outside_allow(glob matching reusing the existing path-glob engine, extended to also match a whole matched subtree).AgentDefinitiongrowsfield_allow/field_deny(schema:repos.fieldAllow/repos.fieldDeny), with aBASELINE_DENY_FIELDSan agent can never remove (extraManifests,networkPolicy,httpRoute,release,releaseTracking,rollout.image- the same fields AGENTS.md already named by convention). Wired intopolicy.pyas a new ruleR020, given a file’sbeforeand proposedcontentin therepo.pr.open/open_uppertool args - a file with nobefore(a new file) is exempt, since there’s nothing to diff against. 235/235 tests pass, including 5 new end-to-end R020 cases and 8 newscope.pyunit tests. - AF-7a, done,
airframe(branchaf7a-walkthrough-runner).docs/walkthroughs/*.yaml(schema in that directory’s ownREADME.md): each step has acommand(documentation), anexpectedresult (prose), and averify(a real, idempotent shell command against the live clusters - exit 0 = still true).tools/walkthrough-runnerreplays a walkthrough’sverifysteps and reports PASS/FAIL, or--renders the command/expected text to markdown. Converted all 3 quickstarts (boarding-api,flight-api,skyport-broker) - all three replay green for real against dev/prod right now (7/7, 4/4, 3/3 steps), satisfying this milestone’s own exit line. Found and fixed a real, unrelated bug along the way:hangar/tools/airframe-scorecard/scorecard.py’s XRD glob matched*.meta.yamlsidecar files too (added by AF-3), crashing the whole scorecard with aKeyErrorsince SP-3 addedmongodb.meta.yaml- nobody had run it successfully since. Fixed; scorecard now runs clean (57.9/100, “Docs for agents” 100/A+, walkthroughs check passing). - SP-4 (OAuth component,
skyport-auth), done and live-verified end to end,airframe#17/#19-#22. Dex (dexidp/dex) chosen over Authentik (lighter footprint, already running in-cluster as ArgoCD’s own SSO).mode: attachregisters a client via Dex’s real gRPC Admin API (CreateClient) through a new custom Crossplane Function,function-dex- this catalog’s second hand-written Function, afterfunction-rollout-watcher. Real, live proof, not offline: the realskyport-authserver (running on dev, kept as the permanent component, not torn down) issued a realclient_credentialsaccess token to a real registered client, and the JWT’s signature was validated against the real JWKS endpoint with PyJWT.- Solved multi-arch for real using the
containerCLI (no docker/podman machine) +skopeofor image-format conversion;crossplane xpkg push -f a.xpkg,b.xpkg TAGgenuinely combines two single-arch builds into one real multi-platform manifest - confirmed viaskopeo inspect --raw. Full recipe infunctions/function-dex/README.md. - Known, accepted tradeoff:
client_credentialsisn’t in any stable Dex release yet (confirmed absent from v2.45.1, the latest as of 2026-09-29) - only on Dex’s unreleasedmasterbranch. Pinned toghcr.io/dexidp/dex:master(a floating tag, no version stability), user-confirmed given there’s no alternative with the feature at all. Revisit once a stable release ships it. Resolved 2026-10-09: Dex v2.46.0 ships the grant (enabled by listing it inoauth2.grantTypes, which the config already does); function-dex v0.1.7 pins it (airframe#79), live on dev with a token verified against JWKS. - Three real bugs found and fixed live before this worked (Dex refuses to start with zero connectors; a wrong healthz probe path; four resource kinds with no
status.conditionsforfunction-auto-ready’s generic detection), plus a Crossplane RBAC gap (PersistentVolumeClaim, user-confirmed) and a Dockerfile portability fix (containerCLI doesn’t support BuildKit’sRUN --mount=target=.). Full writeup:project_sp4_oauth_dex_investigation.md(session memory).
- Solved multi-arch for real using the
- The
platform/→airframe/rename (AF-5’s own step 4) is scoped as a separate follow-on, not part of M2’s AF-5 line item above - it touches every live app’s file layout, both clusters’ ApplicationSets, ~10 Glidepath Tasks, Tower’s Config tab, and every scaffold template, and the design doc’s own plan is a dual-path migration window, not a drop-in rename.
M3 Autopilot core (weeks 11-18), target 88
| Task | Notes | Status (2026-11-02) |
|---|---|---|
AP-A1 apron components.autopilot toggle, registry autopilotReady, network-policy canary |
refused on type: upper; the canary must pass before autopilotReady flips (dev enforces NetworkPolicy: Cilium, verified 2026-10-09) |
Done 2026-10-09; re-attested on hangar-dev (autopilotReady: true in its registry) |
AP-A2 Clearance’s runtime gateway deployed on hangar-dev from the clearance repo, as an Airframe app; a durable session store; audit written in the draft evidence schema; T0 tools with real adapters; deletes each finished AgentRun after recording its evidence (D20; the backstop reaps at deadline + 60 min) |
Backstage MCP, ArgoCD read-only account; built here (owner, 2026-10-28) | Done, live on hangar-dev since 2026-10-29; people sign in through people-dex (D19 interim); catalog.read and argo.sync.lower live; platform-system/clearance retired |
AP-A3 AgentRun XRD, function-agentrun, provider-kubernetes grants, namespaced Role |
copy-tested; expiry proven with Clearance stopped and with the function pod killed | Done, live: AgentRuns in autopilot-runs on agent-sandbox v1.0.8, function-agentrun, the reaper CronJob |
AP-B1 interceptor /agent-installation-token, the agents’ own GitHub App (D4), ArgoCD accounts |
commit signing executed (D1, Hangar root CA); every agent PR and commit carries task_id and tool metadata, so Clearance’s provenance detector can say confirmed agent |
Done for the route and the agents’ App (hangar-autopilot-agents, PRs through the broker); commit signing (D1) not started; no Argo CD account needed |
AP-B2 T1 tools with real adapters; agent-scope and agent-identity gates, stub then real |
field-aware, from AF-9a | Done, live: agent-gates- required; T1 tools real (repo.pr.open, glidepath.cicd.propose, argo.sync.lower) |
AP-B2c clearance-api: Clearance’s read API as a separate read-only process (sessions, run trees, evidence with chain verify, receipts, agent definitions, the inbox feed), roles person/delegate/notifier/auditor, a generated contract Tower vendors (spec, owner 2026-10-30) |
phase 1 done when Tower’s Agent requests poller uses the feed; then Tower’s Clearance plugin, then SSE events | Done: clearance-api v0.13.0+, two replicas; Tower’s agent-requests poller reads the inbox feed; the Clearance plugin is built |
| AP-B3 model proxy v0 (HTTP forwarder), hosted model route, key via Infisical | Done, live: the model proxy to Anthropic, streaming since clearance v0.26.0 | |
AF-8a read-only airframe.* tools: capabilities, validate, explain, plan, status, describe, verify |
first in M3 (owner, 2026-10-28); Kubernetes reads as platform-system/clearance with a token minted per call, never Secrets |
Done 2026-10-28; airframe.validate is not in the gateway image |
AF-8b airframe.apply (opens PRs as the agents’ App), planner adapters |
the parachute sentence passes live, with a seeded bad run and a resume test | Not started |
AF-6b describe and verify contracts; AF-10c chart agent: block |
not rechecked | |
AP-C1 Flight recorder: task_id everywhere; each run records the pinned model version and its Preflight result |
records in Clearance’s evidence schema (verified versus claimed, coverage and assurance state). The anchor is the Clearance ledger (Tessera plus a witness, Clearance roadmap phase 2), not a local checkpoint | not rechecked |
| AP-A4 stable agent identity, separate from a run (per-agent policy, history, memory scope) | one identity per AgentDefinition; runs inherit it. The identity is an ID-JAG (RFC 8693 exchange at the IdP, then RFC 7523 bearer to Clearance; design.md 0.1); R001’s principal is its subject. Blocked on an IdP that issues ID-JAG (D19) | Blocked on D19 (an IdP that issues ID-JAG) |
| AP-C2 cost accounting: tokens and compute per run, per agent, per task tree, in the recorder and the Tower tab | from budgets already enforced | not rechecked |
AF-2b fix the T0 read set (getLogs, getMetrics, getRelease, getDeploymentStatus) and put it in the contract bundle |
read tools taught by example | not rechecked |
Exit: the parachute sentence works end to end on dev (a human merges the flight PRs); killing Clearance mid-run still ends every run on time; the seeded bad run scores as failed.
Clearance and Autopilot build: W0 and W1 (2026-11-03). The build spec’s first two waves are done, and the W1 gate
(“installs alone”) passed on 2026-11-03. The M3 table above is as of 2026-11-02: where it says the gateway runs
from the clearance repo, read the rows below.
| Piece | Status |
|---|---|
W0: R-1 the Go core, provider protocol and conformance harness; R-6 clearance.evidence/v1; R-8 CHANGELOG, release workflow and docs (clearance#42, #43; autopilot#68; hangar#108, #109, #110) |
Built, merged 2026-11-03 |
R-2 Clearance’s own image, chart and Go clearance api (clearance#44) |
Built, released in v0.27.0 |
| R-3 the AI gateway moves to Autopilot: the Python moved with its history (autopilot#69), its own read API (#70), calls to the core with its own database (#71), the core of v0.28.0 (#72; core side clearance#48) | Built, in gateway-v0.4.0 |
R-5 policy bundles (clearance.policy/v1); Hangar’s vocabulary out of the core and into configuration (clearance#49, autopilot#73); decisions D23 and D24 |
Built, in clearance v0.29.0 and gateway-v0.4.0 |
R-7 Clearance as its own apron component, 55-clearance; people-dex moved to 55-identity (apron#151; apron#152 lets the core and clearance-api reach the API server on Cilium) |
Built, live on hangar-dev |
| R-4 the cutover on hangar-dev (gitops-hangar-dev#107) | Built, live since 2026-11-03 |
R-3d the Python package leaves the clearance repo; the core is Go only (clearance#50) |
Built, released in v0.30.0. hangar-dev still runs 0.29.0: apron pins chart 0.29.0 |
What runs on hangar-dev, in autopilot-system:
- The Go core, clearance v0.29.0 from the clearance chart 0.29.0 (image
ghcr.io/jfillman/clearance@sha256:afdca0b4…48dd4):clearance-core(one replica, the only writer ofclearance-pg) andclearance-api(two replicas). The core loads the Hangar policy bundle with its live glidepath-api source, andCLEARANCE_TRUSTED_REGISTRIESisghcr.io/riley-roooo/(D23). - The gateway, gateway-v0.4.0, which calls the core for every decision and record. It has its own database,
autopilot-pg, and its own read API,autopilot-api(the oldclearance-api, renamed). The gateway’s Deployment is still namedclearance.
The W1 gate (passed 2026-11-03).
- Installs alone: the release workflow’s install test passes on a bare kind cluster (no Airframe, no Glidepath) with the published image, for every release from v0.27.0 to v0.30.0.
- The chain is unbroken across the cutover. The Python gateway’s last record was seq 640, head
07e7a9d8…. The Go core opened the same store with the same head and continued at seq 641. The Go chain verify is ok, anchored at 641:07e7a9d8. - The runtime tables were copied into
autopilot-pgby an in-cluster Job: 24 sessions, 2 human requests, 145 events, 2 results, 1 definition. - R-4’s live evidence, seq 641 to 681:
- 641 to 649:
clearance registerfrom the scratch repo, with the GitHub App’s deployment protection rule approving (641, 642); GitHub App check runs (643 to 646); an agent-gates verdict (647, gitops-backstage#244). - 650 to 665: a researcher run.
- 666 to 667: a hold and its release.
- 669 to 674: a human request.
- 675 to 676: session end.
- 677 to 681: a coding-agent run.
- 641 to 649:
- A streaming model call was not checked on its own; the owner accepted it as covered indirectly.
- A hand-applied fix during the cutover (a CiliumNetworkPolicy for TokenReview) is now apron’s (apron#152).
Owed from W1:
GET /v1/evidence/verifythrough the API.- The apron pin to v0.30.0.
- R-4 cleanup: the runtime tables left in
clearance-pg,GATEWAY_API_TABLES=clearance, the/mcpalias, and the gateway’s rename together with airframe.
Next: W2, gate cores (not started; needs the owner’s go).
- Clearance: C-1 the clearance token; C-6 outage policy, coverage states, break-glass and exceptions; C-8 the policy engine’s verdict in the record; C-9 the Traefik Hub adapter; C-10 clearance-api for the gates.
- Argo CD gate: G-ARGO-1, the design pass and spikes. Design recorded 2026-11-03 (clearance
docs/gate.md, “Phase 2 design pass”; D25 decided by the owner, D26 to D29 accepted 2026-11-03), every spike run on hangar-dev in Warn mode; the two hangar-prod measurements (Kyverno under prod load, the cross-cluster identity) wait for the owner (clearancedocs/spikes/g-argo-1/prod-plan.md). - Terraform gate: G-TF-1, the
terraform-plansubject in the core; G-TF-2, hold, release and outcome on the core API. - Autopilot: A-1 runtime provider conformance and records; A-4 Flight recorder fields and cost accounting (AP-C1, AP-C2); A-5 trigger durability; A-7 runtime images and run polish.
The W2 gate: signed tokens issued; the grant feed served; both subject kinds registered; the Argo design pass recorded with its spike results.
M3 status (2026-11-02). The runtime boundary is built and live on hangar-dev: clearance v0.26.1 in the
autopilot-gateway image gateway-v0.3.27 (apron pins it by digest), clearance-api, CNPG clearance-pg, people-dex,
AgentRuns, the GitHub App Hangar Clearance, and every gateway flag on. The status column above is checked against
each repo’s origin/main. The exit test (the parachute sentence live, Clearance killed mid-run, the seeded bad run
failing) still waits on AF-8b. The 2026-11-02 build spec (Clearance and Autopilot, R/A/C/G packages) moves the AI
gateway’s code from clearance into autopilot (R-3) and Clearance’s core to Go (R-1); the rows above say what exists
today, not where it will live.
AP-A1 done (2026-10-09).
- Registry:
airframe.autopilotReady(airframe#81, v0.3.138). The chart refuses it on any cluster without the control-plane role; CI has a must-fail fixture. dev’s registry runs v0.3.138 (gitops-cluster-dev#63, synced, ConfigMaps byte-identical). - Canary:
autopilot/tools/netpol_canary.py(autopilot#8) renders a run’s real policies withcompose()and pairs every denial with a no-policy control. dev: 16/16 pass. Setting the flag on kind-dev is gitops-cluster-dev#64, left for a person to merge and sync (the registry’s manual sync is the attestation). - apron:
components.autopilot, refused off the control-plane cluster and without the full catalog;55-autopilot/with theautopilot-systemandautopilot-runsnamespaces (apron#21), adopted on dev (gitops-cluster-dev#65, Synced/Healthy; a non-restricted pod is refused there). - Not in AP-A1: the
hubandinferencecluster types (Modelplane, M5). - AP-A2 (Clearance as an InfraService) was to be built separately in the user’s AI lab. Superseded 2026-10-28: it is built here.
M3 status (2026-10-28). Scorecard 89.5, 11 of 14 (baseline-2026-10-25.json): above this milestone’s 88, but the exit
test is the parachute sentence live, and nothing on that path runs yet.
- Clearance is now its own product (Clearance Brief, Architecture and Roadmap, 2026-10-27/28; what it changes here is
design.md section 0.1). It has two boundaries: the runtime gateway, which is what M3 builds, and a deployment gate
on its own roadmap (Nov 2026 to Dec 2027, with go/stop gates). The gateway gets built whatever those gates decide: it is Hangar’s AI
gateway. Its code moves to a
clearancerepo (D16), along the split line in design.md 0.1, which the owner accepted on 2026-10-28. - AP-A2 is built here. The owner will not build it in the lab (2026-10-28). Nothing of it is deployed:
server.pyis stdio, every backend but glidepath-api is a fake, and sessions live in memory.platform-system/clearanceexists on hangar-dev (gitops-hangar-dev#35), but only as glidepath-api’s reader identity. Repo split done 2026-10-29 (D16): autopilot#23 made the package splittable (a core that imports nothing from Hangar; packs for tool policy,clearance.packs.hangarholdingglidepath.*/airframe.*, R021 to R023; handlers and the AgentRun run backend registered by Autopilot’shangar_gateway()), then the core moved with its history to https://github.com/jfillman/clearance (v0.1.0, 180 tests), and autopilot#24 pins it. Clearance policy changes are now a clearance release plus a pin bump. Next in AP-A2: the HTTP transport with interim sign-in (TokenReview for run pods, a Dex token for people, until D19), the durable store (Postgres) in the draft evidence-record shape, the real AgentRun and read backends, and the deployment on hangar-dev. Deployed and live-verified on hangar-dev, 2026-10-29. Clearance runs inautopilot-systemasautopilot-system/clearance; that namespace was chosen because function-agentrun’s egress allows only it.- Built in clearance v0.2.0 to v0.4.0: streamable-HTTP MCP with TokenReview (audience
clearance) and OIDC sign-in; the Postgres store, with sessions and evidence in one transaction and a single writer; the draft evidence envelope;Gateway.expire()(D20). - Built in autopilot (
autopilot-gateway, gateway-v0.2.0, autopilot#25/#26): the real AgentRun store and the T0 reads.argo.app.getreads the Application object through the GET-only Kubernetes reader, so no Argo CD account or token is needed.metrics.queryqueries Thanos andlogs.queryqueries Loki. - Losing Postgres ends the process (exit 70) and a restart reloads.
- Deployed by apron#95/#97/#98 and gitops-hangar-dev#58/#60/#61: a CNPG database and ingress NetworkPolicies (agent-run namespaces to the gateway; the gateway and the CNPG operator to Postgres).
- Verified live:
- auth: no bearer and an API-audience token both get 401;
- denials R001, R006 and R002, the breaker surviving a restart, and the chain continuing across it;
- an AgentRun created and then expired as Clearance’s own identity (namespace and Sandbox rendered, then deleted);
- the read tools through MCP;
- the NetworkPolicies: an agent-run namespace reaches the gateway, any other namespace is refused, and Postgres is reachable from the gateway only;
- Postgres pod deleted: the gateway exited 70 at the next sweep, restarted once, and the open session kept its spend.
- Not done in AP-A2:
- People’s OIDC sign-in. The code is built, but hangar-dev has no IdP for people: its Dex servers issue only client_credentials. This needs an owner decision (which IdP; see D19) and a GitHub OAuth App the owner creates. Until then, the delegated agents (glidepath., airframe., coding-agent) cannot be opened through the deployed gateway.
catalog.read: Backstage runs on hangar-prod only, and its API needs a credential.app.api.get: no definition uses it.argo.sync.lower: a T1 tool, part of AP-B.- Retiring
platform-system/clearance: apron#99 and gitops-hangar-dev#62 are open, plus a manual cleanup after the merge.
- Built in clearance v0.2.0 to v0.4.0: streamable-HTTP MCP with TokenReview (audience
- Agents get their own GitHub App (D4, reconfirmed 2026-10-28). It is a different App from Clearance’s provenance App.
- AF-8 splits. AF-8a covers the read-only
airframe.*tools (capabilities,validate,explain,plan,status,describe,verify), built on the patternglidepath.*proved (autopilot#16/#17), and runs before AP-A3 (owner 2026-10-28). It doesn’t need AgentRun: the agent is a Claude Code session talking to Clearance over MCP. AF-8b isapply, after AP-B1. AF-8a DONE 2026-10-28 (autopilot#19): seven T0 tools over a GET-only Kubernetes reader (path allowlist; Secrets refused before any request), R023 app scope, the planner’scicd.yamlpatch split so the agent half passes R021 and the Flight half is a human-merge step. Live smoke through MCP on hangar-dev passed (96 requests, all GETs on the allowlist). Scorecard 97.1, 13 of 14 (baseline-2026-10-28.json); the last check waits onfromService. - Scorecard A+ (2026-10-29): 97.3, 14 of 14 (
baseline-2026-10-29.json).env[].fromService(airframe#129, v0.3.168) withserviceNamespacePatternpassed by the platform on both clusters (glidepath#176, gitops-hangar-dev#42, gitops-hangar-prod#13, apron#78) removed the last two literals (boarding-api#56, gitops-boarding-api#103, renders byte-identical, live). This is readiness, measured; M3’s exit (the parachute sentence live, Clearance killed mid-run, the seeded bad run failing) is not met. Next: AP-A3. - AP-A1’s attestation was lost in the hangar-dev rebuild (2026-10-21). The
autopilot-systemandautopilot-runsnamespaces exist again, butairframe.autopilotReadyis not in hangar-dev’s registry. Re-runtools/netpol_canary.pythere and re-set the flag before AP-A3 creates a run. - Done since 10-09 and used by M3: six
glidepath.*T0 tools with a real HTTP adapter and R022, live through MCP on hangar-dev;cicd.yamlis scoped by field (R021) instead of denied whole (D13). - Scorecard work riding with AF-8a: checks 3 (strict objects: mark the documented passthroughs
x-hangar-passthrough, and stop counting typed maps andif/thenbranches as open objects), 7 (boarding-api’s two cross-appFLIGHT_API_URLliterals need a service reference besidefromComponent) and 11 (the tool check counts registered, tested tools, not a file name). Also: the hard-coded “escape hatches denied by default” score becomes measured, since/extraManifestsis inBASELINE_DENY_FIELDS. Projected 97.2 and 14 of 14 if all of that lands, which is a thin margin.
M3 status (2026-10-09, before AP-A1): not started. Nothing of it existed on a cluster: no AgentRun or agent-sandbox CRD on
dev, no Clearance deployment, no components.autopilot in apron. What moved between M2 and here:
- The Tier 2 rename finished on 2026-10-01:
catalog.hangar.iois the only catalog group, and the autopilot drafts emit it. - The 2026-10-08 architecture review closed on 2026-10-09 (airframe v0.3.137 on both clusters). It leaves A4
(condition vocabulary,
observedGeneration) and A6 (XR rule ids) to theairframe.*tools (AF-8), and A5 (Holmes through Clearance) to AP-A2. - The planner follows ADR-0019 (
deploy.environments,glidepath/envs/, the tenants repo from the registry; autopilot#6). autopilotmain: 240 tests pass. - The AppSpec schema now lives in Airframe’s contract bundle (
contract/appspec.schema.json, airframe#80). Autopilot vendors it and its CI fails on drift (autopilot#7). - apron’s cluster template is on airframe v0.3.137 with what that catalog needs (apron#20). AP-A1 builds on it.
- Scorecard 86.6, 11 of 14 checks (
baseline-2026-10-09b.json) against this milestone’s 88. The lowest dimension is the interaction surface (54.5), which waits on theairframe.*tools (AF-8).
Decisions before M3 code (answered 2026-10-09 unless marked open):
cicd.yamlis agent-editable. The CI/CD config is part of the app, so an agent allowed to configure the app may configure it.cicd.yamlleavesBASELINE_DENY_PATHS;.tekton/**stays denied. These stay human-only (decided 2026-10-09):- Field-level deny (AF-9a
BASELINE_DENY_FIELDSforcicd.yaml):/governance(the scan gates; an agent adding itself toallowedCommitSignerswould have approved its own release - since 2026-10-20, glidepath ADR-0027, the signer list is admin-owned catalog values andcicd.yaml’s copy is ignored, so the deny is now belt and braces),/deploy/releaseFile,/deploy/chart(another chart renders anything, past the schema and chart guards),/deploy/targetwith/deploy/ecs,/deploy/lambda,/deploy/azureContainerAppsand the same blocks on each environment (they point the cloud deployer elsewhere), and/secrets(withbuild.scripteditable, an entry is a path to exfiltrate any key in the app’s store). - A new Clearance rule matching environments by name, because the pointer diff collapses a list-length change to
the whole list: an existing environment’s
tier,productionandclustercannot change, and flight environments cannot be removed or reordered (the order is the promotion order). Adding environments stays allowed. - Pipeline steps: no
gitops-image-bumpstep (itsgitopsRepo/manifestPathreach any repo); noreleasestep to aproductionenvironment in a flow with an automatic (push or event) trigger. Other release steps stay allowed, since the planner adds one for the first flight environment. - Allowed, with notes:
build.script/containerfile/unitTest.command(check the build step holds no credential the unit-test step lacks, such as registry push),ephemeralEnvironments(cap the TTL),notifications, triggers.
- Field-level deny (AF-9a
- dev enforces NetworkPolicy. It runs Cilium 1.20 with Kubernetes NetworkPolicy on; a scratch test showed ingress and egress deny-all enforced and a pod-selector allow restoring traffic. The AP-A1 canary is expected to pass there; the earlier “fails by design” premise was never tested.
- The U8 ClusterRole is approved (
autopilot/airframe-drafts/rbac/provider-kubernetes-agentrun.yaml, 2026-10-09). Review notes for AP-A3: the grant lands on provider-kubernetes’s shared service account, so it covers every provider-kubernetesObject, not only function-agentrun’s; Job and ServiceAccount create in any namespace can run a pod as any existing service account; namespace patch can drop pod-security labels or add the PR-sweep label anywhere. Approved to ship together with it (2026-10-09): a ValidatingAdmissionPolicy matched on the provider’s service account that confines these kinds toagent-r-*run namespaces carrying the run label, never touchespod-security.*labels or adds the PR-sweep label, and allows onlyserviceAccountName: run; thebatch/jobsrule swapped for the agent-sandbox resources (U11); a binding through a stable service account name. A test proves the policy rejects a Job indefaultand a label patch onkube-system. - Holmes stays at 0 replicas, on purpose. A5 (Holmes through Clearance) waits until Holmes is wanted again.
- D1: a new Hangar root CA for the platform’s Fulcio. Agent commits (and the platform’s other keyless signing) chain to a Hangar-owned root rather than a per-cluster self-generated one. Design and rollout are not started.
Gaps in the existing code that M3 closes: nothing maps a run pod’s identity
(system:serviceaccount:agent-r-<id>:run) to the session principal R001 compares against (AP-A4, now by ID-JAG: the run
exchanges that token at the IdP, and the principal is the ID-JAG’s subject); sessions are held
in memory and their audit entry stores the requested limits only as a hash, so a Clearance restart loses the grant
(AP-A2 needs a durable session store).
Build order (revised 2026-10-28): AP-A1 (done; re-attest on hangar-dev) -> AF-8a with AF-6b’s describe/verify and the
scorecard fixes -> AP-A3 (render an agent-sandbox Sandbox, per U11; hard expiry from its shutdownTime, per U2) -> AP-A2
(the clearance repo split, then the gateway deployed, with AF-2b) -> AP-A4 -> AP-B1 -> AP-B2 -> AP-B3 -> AF-8b with AF-10c
(the parachute run) -> AP-C1 and AP-C2. The previous order put AF-8 after AP-B3; its read-only half needs none of those.
Previous order (2026-10-09): AP-A1 -> AP-A3 -> AP-A2 with AF-2b -> AP-A4 -> AP-B1 -> AP-B2 -> AP-B3 -> AF-8 with AF-6b and AF-10c -> AP-C1 and AP-C2.
M4 Skyport AI workloads (weeks 19-24), target 93
Parts 6 to 11, in shape order: task, session, service, scheduled, event, team. Each is walked live, has a Preflight case with a seeded bad run, and updates its quickstart’s verified-state note.
| Task | Notes |
|---|---|
Part 6 flight-briefer |
first agent; proves the whole path |
Part 7 gate-copilot |
session channel and Tower’s chat and approvals |
Part 8 passenger-assistant |
the hardened-sandbox fail-closed moment; injection test |
Part 9 delay-digest |
trigger runner; artifact store |
Part 10 disruption-responder |
trigger bridge; redelivery and storm demos |
Part 11 irregular-ops-team |
narrow-only, a denied spawn, a checker catching a wrong fact |
A new, standalone Autopilot Backstage plugin (fleet-wide console: runs, budgets, the breaker, Preflight, audit) and a narrow Tower Agent tab for app-scoped session chat, reusing the plugin’s backend the way Tower’s Glidepath tab reuses Glidepath’s (decided 2026-09-27, design.md section 8 item 3a); Agent XRD and two templates; Holmes via Clearance (drop github-mcp-token) |
Exit: six Preflight cases pass and their seeded bad runs fail; one task_id traces a disruption from broker message to final artifact.
M5 Evidence and widen (weeks 25-28), target 97 and 14 of 14
- Preflight as a regression gate on the Autopilot repo: profile, model and policy changes are PRs that must pass.
- Autonomy tracker for one alert class (shadow, then approve, then auto), with the breaker.
- Modelplane hub trial on a dedicated kind cluster, behind the OpenAI-compatible contract.
- AP-D1 opt-in persistent, resumable state for session and team shapes: size cap, retention TTL, scan before restore (needs U12; design in design.md 3.1).
- Part 12: the nginx edge (optional), born A+.
- Conformance on prod: upper clusters get no runs, and a run cannot be created there.
- The A+ run: scorecard 14 of 14 and overall at least 97, recorded.
Clearance build: W4 gate (closed 2026-11-09)
The owner closed the W4 gate of the Clearance and Autopilot build specification on 2026-11-09, with one check postponed.
- Live packet: built by Clearance 0.36.0 from the hangar-dev database: 1,266 records, covering 2026-10-29 to 2026-11-09. It verifies offline with the log key, witness policy, trust root and signer identity supplied separately: record hashes and chain, log signature, the hangar-prod witness’s countersignature, 1,266 inclusion proofs, and 7 keyless attestations.
- Negative controls: one altered record fails the manifest, chain and inclusion checks; a wrong witness key fails the witness check.
- Coverage statement: carries the six coverage states and states observation is blind, since no collector exists until W5.
- Conformance: the Argo CD and Terraform providers both pass.
- Postponed: the 30-day window. The chain began on 2026-10-29 and the first countersignature was on 2026-11-09. Re-run the packet on 2026-12-09, when 30 days of records have each been witnessed as they happened.
3. Critical path
chart guard → validate → contract → ownership split → Autopilot core → planner (parachute) → Skyport AI.
Parallel and safe to interleave: Skyport components (born A+), docs and walkthroughs, the Autopilot
repo setup, Modelplane reading. Do not parallelize the ownership split with Clearance write tools.
4. Decisions
| # | Decision | Recommendation | Needed by |
|---|---|---|---|
| D1 | Commit signing for agent PRs | sign with the self-hosted Fulcio; extend the provenance gate; do not exempt agent PRs |
M3 |
| D2 | AgentRun as an XR | yes (not a bare Job, not a controller) | M3 |
| D3 | Clearance standalone or a Backstage module | standalone, federating Backstage MCP. Since 2026-10-28 a product of its own (D16) | M3 |
| D4 | A separate GitHub App for agents | yes, so agent traffic cannot starve CI of API budget. Reconfirmed 2026-10-28; not the Clearance provenance App | M3 |
| D5 | Audit store | enable MinIO object lock when the bucket is created. Revised 2026-10-28: integrity comes from the Clearance ledger (hashes in a Tessera log, a witness Clearance does not run); object lock is retention only | before AP-C1 |
| D6 | Hub isolation for Modelplane | Decided 2026-09-26: hosted providers only for now (option 1); per-environment hub with dedicated accounts (option 2) is the M5 default | before M5 |
| D7 | Preflight’s name | Airworthiness (open; the user likes Autopilot, Clearance, Flight recorder) | before AP-B2 |
| D8 | A new autopilot repo |
yes; clearance/ becomes a package in it |
M0 |
| D9 | Release-file split and the ArgoCD exclude |
prove on a scratch app first | M2 |
| D10 | Strictness rollout dates | Decided 2026-09-26: option A, enforce now (the fleet sweep is clean). airframe validate becomes a required check on gitops PRs. Built 2026-09-26, not yet required: Airframe publishes ghcr.io/jfillman/airframe-validate (multi-arch, on release tags; contract validate-values), Glidepath has the generic validate-values Task, values-check Pipeline and a gitops onboarding template. Rollout: resync PRs deliver .tekton/pull-request-values.yaml to each gitops repo (baggage-api’s is #3; boarding-api, flight-api and skyport-broker were triggered), then register values-validation in releaseGuardrails to make it required. Ground (app repo) env edits are not gated yet: direct commits to main have no PR to block |
M1 |
| D11 | Debugging a run without pod exec | accept: debug via output, logs and audit | M3 |
| D12 | Where agent code lives | airframe/examples/skyport/agents/ |
M4 |
| D16 | Clearance’s repo | Decided 2026-10-28 (Clearance architecture): its own clearance repo; supersedes D8 for the package. The split line (design.md 0.1) was accepted on 2026-10-28 |
AP-A2 |
| D17 | Runtime boundary: Clearance as an AuthZEN PDP behind Traefik Hub’s Sovereign Trust Plane, or its own gateway | Open. Recommendation: PDP, own gateway as fallback. Hangar runs Traefik Proxy v3.7.6, not Hub (verified 2026-10-28), so the own gateway ships now, with the decision refactored into a pure, AuthZEN-shaped function | before AP-B2 |
| D18 | AF-8a before AP-A3 | Decided 2026-10-28 | M3 |
| D19 | Which IdP issues agent ID-JAGs | Open. The standard is decided (ID-JAG, owner 2026-10-28). Dex cannot issue ID-JAGs today (dexidp/dex#4611 open; v2.46.1 lacks it). Options: wait for and pin a Dex release that ships it (recommended, since Dex is already Hangar’s IdP); run a second IdP that issues them (Keycloak documents ID-JAG in its nightly docs; unverified in a release); a Clearance-side issuer (rejected: it is the decision point, so it must not mint its own principals). AF-8a does not need it: it reads as platform-system/clearance |
AP-A4 |
5. Unverified: answer these in M0
| # | Question | How to check |
|---|---|---|
| U1 | Does ArgoCD apply XRs so that unknown fields are rejected, or pruned silently? | apply an XR with a bogus field to the dev cluster via a scratch Application |
| U2 | Does the function response TTL re-invoke function-agentrun near expiry? |
a toy composition on a scratch namespace; the expiry test in M3 depends on it |
| U3 | Is Backstage’s mcpActions endpoint live on a cluster? |
request it from the Tower backend |
| U4 | Were Tower’s Tier 1 write actions ever live-verified? | click Refresh, Sync and Re-run against a genuinely stuck app |
| U5 | Does the ArgoCD glob *-lower/* match <app>-lower AppProject names? |
argocd proj list against the policy |
| U6 | Does Helm ignore unknown x-hangar-* keywords in values.schema.json? |
helm lint on a schema with the keys |
| U7 | Can the ArgoCD git files generator exclude *.release.yaml? |
a scratch ApplicationSet |
| U8 | Does provider-kubernetes accept the rendered Namespace, Quota, NetworkPolicy, ServiceAccount and Job? | crossplane render is not possible without Docker; use a scratch XR on the dev cluster |
| U9 | Is pgvector available in the CloudNativePG image? | only if a vector store is wanted |
| U10 | What do Modelplane’s usage records contain, and does short-lived auth work for Existing clusters? |
read a real record, in the M5 trial |
| U11 | Is the upstream Kubernetes SIG Agent Sandbox real and mature enough (warm pools, gVisor/Kata, suspend/resume)? Could function-agentrun render its Sandbox object instead of a raw pod? |
read the project and its CRDs, then install it on the dev cluster and render one run through it; answer before AP-A3 is built. Its current maturity is second-hand knowledge (see industry-context.md) |
| U12 | Can a run’s workspace be snapshotted and restored (volume snapshot on the dev cluster’s storage class), and what does restore cost in seconds? | scratch PVC plus VolumeSnapshot on the dev cluster; needed for AP-D1 |
All U-experiments run on the dev cluster (decision 2026-09-26).
Answers (2026-09-26)
| # | Answer |
|---|---|
| U1 | Pruned silently, by default. ArgoCD synced a Redis XR with persistance: true and bogusField: 1, reported success, and the stored spec had neither. A server-side dry-run with --validate=strict rejects them (strict decoding error: unknown field), warn prints a warning, ignore accepts. Enum values (size: gigantic) are rejected by the XRD schema either way. So typos in an XR file look accepted and do nothing: airframe validate must cover XR files too. |
| U5 | Yes. argocd admin settings rbac can with p, role:clearance-lower, applications, sync, *-lower/*, allow: flight-api-lower/... and backstage-lower/... allowed; flight-api/... (upper) and app-flight-api-cicd-pr/... denied. |
| U6 | Yes. helm lint and helm template accept unknown x-hangar-* keys at the root and on a property; type enforcement still works (appName=5 fails). |
| U7 | Yes, with the real name. A git files generator with path: .../*.release.yaml, exclude: true generated exactly the dev and test apps and none for dev.release.yaml (live, scratch ApplicationSet on the dev cluster). |
| U2 | No, not on our Crossplane. A function’s response TTL only takes effect when Crossplane’s beta realtime-compositions flag is on (it is off: the args are core start); otherwise the XR is requeued at the poll interval, default 1 minute, and the per-XR crossplane.io/poll-interval annotation cannot go below --min-poll-interval, default 1 minute. Measured: a composition that renders the current time re-rendered every 60 s whether the TTL annotation or a 10 s poll-interval annotation was set. Also: installed function-go-templating v0.12.3 has no TTL support at all, and installing a second package from the same repo under another name collides in the package lock (health flapped for about a minute, then recovered; nothing was left behind). Consequence: expiry cannot be driven by re-invocation faster than about a minute. Use the Sandbox shutdownTime (U11: enforced within a second) or the Job’s activeDeadlineSeconds for hard expiry, and treat Crossplane’s poll as the status refresh only. |
| U8 | Yes, once granted RBAC; today it is denied. Rendered the six AgentRun objects with function-agentrun’s own compose() (docker.io alpine pinned by digest) and applied them as provider-kubernetes Objects on the dev cluster. Before any grant every one failed to observe with forbidden (provider-kubernetes has no rights on namespaces, resourcequotas, serviceaccounts, networkpolicies or jobs). With a scratch ClusterRole for exactly those kinds, all six synced and went Ready, the Job completed under pod-security: restricted, and deleting the Objects removed the run namespace. The narrow role is drafted in autopilot/airframe-drafts/rbac/; it must be reviewed as a privilege grant (namespace create/delete cluster-wide) before it ships. Not tested: NetworkPolicy enforcement (believed then not to be enforced on the dev cluster; it is, see “M3 status”), the Modelplane path, or the composition running under a real XR (the function image is not built). |
| U3 | Partly. On the prod cluster, Backstage serves /api/mcp-actions (401 “Missing credentials”, where a made-up path returns 404), so the plugin is mounted and behind auth. Not verified: that it lists Tower/catalog actions with a real token. |
| U4 | Yes, live-verified 2026-09-27. baggage-api PR #2 (branch u4-failing-run) produced a real failed PipelineRun; Tower’s Re-run and Cancel were exercised against it and confirmed working before the PR was closed and the branch deleted. |
| U12 | Not with VolumeSnapshot on the dev cluster today. The cluster has no snapshot CRDs and its only storage class is rancher.io/local-path. Workspace persistence for AP-D1 needs either a snapshot-capable CSI driver or an archive-to-object-store approach (tar the workspace, scan it, store it); restore cost is not measured. |
| U11 | Real and usable; adopt behind the AgentRun contract. kubernetes-sigs/agent-sandbox v1.0.4 (2026-09-24, weekly releases, API v1beta1): Sandbox, SandboxTemplate, SandboxClaim, SandboxWarmPool. Installed on the dev cluster (arm64 image works) and measured: a cold sandbox Ready in 2 s (image cached), suspend and resume in about 1 s, a claim served from a warm pool Ready in 89 ms, and a Sandbox with shutdownTime and shutdownPolicy: Delete was deleted by the controller within a second of expiry. SandboxTemplate carries networkPolicy/networkPolicyManagement and an env-injection policy, which overlap what function-agentrun renders. Not tested: gVisor/Kata, snapshot restore, and NetworkPolicy enforcement (believed then not to be enforced on the dev cluster; it is, see “M3 status”). Removed again after the trial. Decision for AP-A3: render a Sandbox (or claim) instead of a raw pod, and treat the controller as a pinned dependency. |
6. Risks
- One person, many repos. airframe, glidepath, backstage, apron, the tenants repos and a new repo all change. Keep every change small, behind a gate, and reversible. Use worktrees.
- Retrofit cost if the contract slips behind the next components. Hence the ordering.
- NetworkPolicy enforcement is per cluster. dev enforces it (Cilium; ingress and egress verified 2026-10-09; an earlier version of this list said it did not, which was never tested).
autopilotReadyis still gated on a canary, so runs are never trusted on an assumption. - The prod cluster is resource-limited (observability scaled to 0). Runs are dev-only; prod gets none.
- Hosted model cost. Token budgets and the breaker; two on-demand or scheduled spenders in the demo.
- Upstream churn. Modelplane is v0.1 and the mcp SDK is v2; both sit behind contracts and pins.
- Strictness breaks an environment. Warn first, sweep the fleet, then enforce.
7. First ten actions
- Read
HANDOFF-hangar-autopilot-airframe-a-plus.mdand check each repo’s branch and status. - Decide D8 (the
autopilotrepo) and D7 (Preflight’s name). - Answer U1, U5, U6 and U7 (quick, no cluster changes).
- Write the chart guard in an Airframe worktree with a fixture test; run it through a scratch app.
- Wire the scorecard and chart tests into CI.
- Commit
AGENTS.mdfor Airframe. - Build
airframe validatev0 (schema plushelm template). - Provision
baggage-apithrough the current flow and log each stumble. - Put
airframe validateonbaggage-api’s PRs. - Re-run the scorecard and update this file’s status.