Docs · Hangar

Autopilot design: running any AI agent workload on Hangar

Autopilot is the sixth Hangar product, beside Tower, Glidepath, Airframe and Apron. This is its design. The plan to build it, and the Airframe work it depends on, are in roadmap.md and airframe-ai-friendly.md. Every claim is labelled Built (exists and is tested), Draft (written, never applied to a cluster), Proposal (designed, not written) or Unverified (depends on something not checked). Code lives in the autopilot repo (decision D8). Clearance’s package moves to its own clearance repo (section 0.1, D16).

Names, checked against the Hangar Brand System on 2026-09-26. Autopilot (a sixth product beside Tower, Glidepath, Airframe and Apron, and the apron component name), Clearance (tool gateway and policy; since 2026-10-28 a product of its own, section 0.1) and Flight recorder (audit and correlation) are accepted. Preflight (evaluation) was decided on 2026-09-26 (D7). Caveat on Flight recorder: “Flight” already means the upper-environment tier in Tower and the Airframe quickstart (Ground and Flight), so define it in the glossary as recording both tiers. Flight recorder is a feature, like Tower’s “Ground Control” and “Release Record”, so it needs no mark of its own. Clearance, now a product, does.

0. Where this stands

0.1 Clearance is a product of its own (2026-10-28)

Sources: the Clearance Brief, Clearance Architecture and Clearance Roadmap (published 2026-10-27/28: brief, architecture, roadmap). Those three are the source for the product. This section records what they change in Autopilot’s design. Where this document and they disagree, they win for Clearance and this document wins for Autopilot.

What changed. Clearance is no longer a feature inside Autopilot. It is a standalone Hangar product. Autopilot depends on it and is its native consumer, and the Hangar lab is its reference deployment. It has two boundaries that write one record:

They share one tier vocabulary, one evidence schema, one ledger and the task_id, so a packet can show the tool call that drafted a change beside the clearance that landed it. Runtime tiers T0 to T3 and deployment tiers 0 to 3 share names, not rules: a T2 tool (propose a PR to an upper environment) produces a change that is tiered again, on its own merits, at the deployment boundary.

The runtime gateway is a Hangar deliverable whatever happens to the product. The Clearance roadmap has stop conditions (no budget owner, an in-house build under two weeks, nobody pays). They decide the deployment-gate business. They do not decide whether Hangar builds the gateway Autopilot needs: Clearance is Hangar’s AI gateway at the very least (owner, 2026-10-28). AP-A2 is built here, not in a separate lab.

What it changes in this design

Repo split (D16, accepted by the owner 2026-10-28, amended by D21 2026-11-02)

D21 moves the line: Autopilot owns the AI gateway, and Clearance decides and records. The gateway enforces, asking Clearance for every decision and writing every record through Clearance’s API, and reaches it no other way.

Where the code is today (Built, 2026-11-03): the code follows the D21 line. R-3 moved sessions, dispatch, the gateway, the model proxy and the MCP surface to autopilot (autopilot#69 to #73), and R-3d removed the Python package from clearance (v0.30.0), whose core is now Go only. On hangar-dev since the R-4 cutover, the gateway (gateway-v0.4.0) calls the Go core (clearance v0.29.0, clearance-core and clearance-api from the clearance chart) for every decision and record, and keeps its own database and read API (autopilot-pg, autopilot-api). See the roadmap’s “Clearance and Autopilot build: W0 and W1”.

The line: Clearance must install at a customer that runs no Airframe, no Glidepath and no Autopilot. Hangar-specific backends would make it a Hangar component, not a product. The tool policy (the rules, the tier vocabulary, scope, R021 and R022) is Clearance’s; the tool registry and implementations are Autopilot’s. Clearance’s own code is Go (D22); the gateway moved in Python (R-3) and A-9 ports it to Go.

Runtime boundary on a standard

Traefik’s Sovereign Trust Plane (GA 30 Sep 2026) does at the request path what the gateway’s dispatch does in Python: delegation by token exchange, inline enforcement through an AuthZEN query that returns a structured denial, and a witnessed log of every decision. Hangar’s rule is to adopt leading technology rather than reinvent it. Verified 2026-10-28: both clusters run open-source Traefik Proxy (docker.io/traefik:v3.7.6, GatewayClass traefik), not Traefik Hub, so STP needs a Hub licence or the feature reaching the open proxy. Until one of those happens:

• Hangar keeps its own gateway: AP-A2 deployed it, and since R-3 it is Autopilot’s (D21).
• Dispatch is refactored so the decision is a pure function, from an AuthZEN-shaped request (subject, action, resource, context) to a decision with a rule id and a hint. That makes moving behind STP a change of transport, not of policy. Denials already return a rule id and a fix hint (section 5.3), which is the structured 403 STP returns.
• The STP evaluation runs against a GA build in the lab (Clearance architecture ch. 14), never early access.

Timing against the Clearance roadmap

The Clearance roadmap starts in Nov 2026 with two weeks of discovery and no code. The clearance repo, the design record and a frozen evidence schema follow by 2026-11-27. Hangar’s M3 does not wait for that, with one rule: anything M3 writes that becomes a record (AP-C1’s Flight recorder fields, the session store’s audit) is built to the draft evidence schema, because changing the schema after records exist is the one migration both roadmaps say to avoid.

Stale numbers in the three sources: they say 254 tests and 15 deny rules. autopilot main on 2026-10-28 has 308 tests and 23 rules, R000 to R022 (R020 field scope, R021 cicd.yaml scope, R022 Glidepath app scope were added since).

Three design decisions shaped it, each made with the project owner:

1. Durable changes are git commits; ephemeral runs are direct requests to Crossplane (section 1).
2. It must run any AI agent workload, not only deploy applications (section 3).
3. Backend infrastructure is managed the Hangar way, with Modelplane for self-hosted models (section 6).

1. The principle, revised

Hangar already answered the hardest question in this space (hangar/docs/service-catalog-design.md §0): no component holds a live Kubernetes write credential, and every mutation is a git commit. That is still right for anything that outlives a run.

It is wrong for a run itself. Committing every session to git would be slow, rate-limited (your GitHub bucket is already shared by four or more tokens and has been exhausted once), noisy in history, and unreviewable, because no human will review a 20-minute sandbox.

So:

Durable changes are git commits. Ephemeral runs are direct requests to Crossplane: a namespaced XR, created through the Kubernetes API instead of through git, and reconciled by an already-privileged and already-audited control plane. The requester never touches the underlying resources.

(Crossplane v2 has no Claims; an AgentRun is a namespaced XR like every other Airframe API. What makes it ephemeral is how it is created and when it dies, not its kind.)

That is still your Tower policy (delegate to a system that already audits itself), and still dev-only.

The ephemerality test

A thing may be created directly through the API instead of as a commit only if all five hold. If any fails, it goes through git.

1. It has a hard deadline that needs no Clearance to enforce.
2. It holds no durable state and reaches the durable plane only as a PR.
3. One narrowly scoped identity creates it, in one reserved namespace.
4. It can only narrow what git grants.
5. It can be rebuilt from git plus the task spec.

What this costs, stated plainly

Clearance now holds one Kubernetes permission: create, get, list, watch, patch and delete agentruns in the autopilot-runs namespace, on dev clusters. That is a departure from “zero write credentials”. It is contained three ways: the permission is one kind in one namespace (a namespaced Role, not a cluster grant), the XRD schema constrains every field, and the composition, not the caller, decides what is created. The caller cannot pass a manifest.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: clearance-agentruns, namespace: autopilot-runs }
rules:
  - apiGroups: ["catalog.hangar.io"]
    resources: ["agentruns"]
    verbs: ["create", "get", "list", "watch", "patch", "delete"]   # patch sets spec.frozen

What stays in git

Agent definitions and policy, profiles and their ceilings, app, gitops and tenants repos, cluster config, the Modelplane hub’s inventory (classes, clusters, deployments, services), model routes, the Preflight corpus, gate definitions. Raising authority is always a reviewed commit. Lowering it is immediate and needs no review.

Industry context: industry-context.md places this design against the emerging “agent substrate” category and records what we adopted from it.

2. Principles inherited, and what each forces

3. The workload model

AgentDefinition (durable, git) says what an agent is: kind, identity, tier ceiling, tools, repos, models, network reach, compute class, limits, image (by digest), framework, sandbox class, sidecars, triggers. Today it is reviewed YAML validated by schemas/agent-definition.schema.json. Later it becomes an Agent XRD, so Tower generates a New Agent form like every other XRD (Phase C).

AgentRun (ephemeral, namespaced XR) is one bounded instance: the definition narrowed by its requested limits, with a deadline.

Services reuse the existing application path, so nothing new is invented for them. Holmes is already one.

3.1 Additions from the substrate review (2026-09-26)

• Stable agent identity. An agent has one identity per AgentDefinition, independent of any run. Runs inherit it. It is the key for per-agent policy, for “everything this agent ever did” in the audit chain, and for any future memory scope. Task: AP-A4.
• Model version is part of the record. Each run records the exact model id/version it used, next to its tools, decisions and Preflight result, so a change in behaviour can be tied to a model change.
• Cost accounting. Tokens and compute are recorded per run, per agent and per task tree, and shown in Tower. Task: AP-C2.
• Optional persistent state (session and team shapes only). Default stays stateless, so a run’s state is its Clearance channel and its artifacts. An agent may opt in to a workspace volume that can be snapshotted and resumed. Guards: size cap, retention TTL, a scan before restore (memory poisoning is the risk), and the volume belongs to the agent identity, never to another agent. Task: AP-D1, gated on U12.
• Sandbox substrate. Before building more pod-rendering code, decide whether function-agentrun should render the upstream Kubernetes Agent Sandbox object (warm pools, gVisor/Kata, suspend/resume) instead of a raw pod. Answer: U11, before AP-A3.
• Rejected: a monolithic AgentWorkspace resource that grants cpu, memory, namespaces and tools in one object. It merges what a run may do with what the platform grants, and it would put kubectl and argocd inside the sandbox. AgentRun plus an optional state volume keeps them apart.

Runtime contract (any framework)

Environment: HANGAR_RUN_ID, _TASK_ID, _SESSION_ID, _AGENT, _EXPIRES_AT, _LIMITS, _INPUT, _OUTPUT_DIR, _WORKSPACE, CLEARANCE_URL, MODEL_PROXY_URL (each URL only if the network mode allows it). Files: /workspace, /run/output, tokens under /var/run/hangar (audiences clearance and model-proxy). On SIGTERM, checkpoint within 30 seconds. That is the whole contract, which is why Claude Agent SDK, LangGraph and a hand-rolled agent all fit.

Sandbox and compute classes

• standard: pod security restricted, non-root, read-only root filesystem, all capabilities dropped, default-deny egress with only the exceptions the network mode names, a quota that forbids Secrets and Services.
• hardened: adds a runtime class. If the cluster has none, the run is Rejected. It is never silently downgraded. The home lab’s local clusters (kind, kiac) have none, so there hardened is always Rejected; apron check refuses to configure one on them.
  • Which runtime: gVisor. It runs on ordinary EC2 instances (its default systrap platform needs no nested virtualization; Kata would need metal instances on AWS), supports arm64 and amd64, and is what Anthropic and OpenAI use for untrusted code. It moved to CNCF governance in November 2026 (see industry-context.md).
  • How a cluster gets it. hangar-infra’s eks-cluster module installs gVisor (version and sha512 pinned) on a node group with gvisor = true and labels and taints it hangar.io/runtime=gvisor; eks-dev has a one-node sandbox group. cluster.yaml autopilot.hardenedRuntime: gvisor makes Apron render RuntimeClass gvisor (handler runsc, scheduled onto those nodes) and give function-agentrun HANGAR_RUNTIME_CLASSES={"hardened":"gvisor"}. One key drives both, so the function never names a class the cluster lacks. The shared Composition’s runtime_classes stays {}.
  • Cost to measure first. gVisor’s I/O is slower than the host’s (the performance patches Google runs are not in mainline Linux), and agent runs do a lot of it: clones, package installs, workspace writes. Benchmark a real run in both tiers before making hardened the default for third-party images. The RuntimeClass declares no pod overhead yet, because overhead counts against the run’s ResourceQuota.
• Compute classes (small, medium, large, gpu) are declared per cluster in the function’s input; a class the cluster lacks is Rejected. This borrows Modelplane’s idea of a platform team publishing hardware classes that workloads request by name.
• Network modes, ordered by reach: none, clearance, clearance+model, allowlist (explicit CIDRs, never /0).
• Agent images are untrusted code. They are signed by Glidepath, referenced by digest (the XRD schema rejects a tag), and admission verifies the signature.

4. Components and repo changes

Status checked against origin/main of each repo and the hangar-dev deployment on 2026-11-02 (clearance v0.26.1, autopilot gateway-v0.3.27, airframe v0.3.172 on hangar-dev).

5. Concrete artifacts

5.1 The AgentRun function: expiry and failing closed

function-agentrun is one pure function, compose(xr, observed, now, cfg), so it is tested without a cluster. Two properties matter most, and both are tested:

• Expiry needs no Clearance. Past spec.expiresAt it renders nothing, and Crossplane garbage-collects the namespace. A dead or compromised gateway cannot leave a run alive. The Job’s activeDeadlineSeconds is anchored to the XR’s creation time, so it does not drift between reconciles.
• It fails closed. A hardened sandbox on a cluster without a runtime class, a compute class the cluster lacks, a TTL over the cluster’s cap, an allowlist of 0.0.0.0/0, or an input over 4 KiB is Rejected and renders nothing.

Lifecycle: Provisioning, Running, Succeeded or Failed (drained after a grace period), Frozen (egress cut, run kept for forensics for a hold period), Expired, Rejected.

It composes only provider-kubernetes Objects, because Crossplane v2 rejects a namespaced XR composing a cluster-scoped resource, the same reason SecretStore wraps ClusterSecretStore. The run namespace must never carry hangar.io/ephemeral-env: the existing PR-namespace TTL sweep deletes namespaces with that label when no owning ArgoCD Application exists, and would reap live runs. A test asserts the label is absent.

5.2 AgentDefinition

See autopilot/agents/*.yaml (five shipped: a task agent, a researcher, an orchestrating team, an event-triggered triage agent, a scheduled reviewer). Baseline deny paths (.tekton/**, cicd.yaml, CODEOWNERS, .github/**, **/appproject*.yaml) apply to every agent and cannot be removed by a definition, so an agent can never edit the controls that constrain it.

5.3 Policy as CEL

Twenty-two deny rules plus R000, first match wins, each with a stable id and a fix hint that goes back to the agent, so it can retry against a deterministic gate instead of guessing. Any evaluation error is R000 and a deny. The rule id plus hint is already the shape of an AuthZEN structured denial (section 0.1).

Tools, by tier: T0 catalog.read, metrics.query, logs.query, argo.app.get, app.api.get, artifact.get, chat.recv, chat.send. T1 repo.pr.open, xr.request, argo.sync.lower, pipeline.rerun, artifact.put, run.spawn, run.close, human.request. T2 (propose only) repo.pr.open_upper. T3 never exposed; the names exist only as tripwires. T0, built: six glidepath.* tools over glidepath-api (contract, runs.list, run.get, logs.get, explain, releases.get). The airframe.* tools (capabilities, describe, explain, plan, apply, status, verify) are the M3 deliverable: the read-only half first (AF-8a), apply after the agents’ GitHub App (AF-8b). The planner behind plan is built.

5.4 Narrow-only, and teams

Requested limits, or a child run, may only narrow what they inherit: tier, time, each budget, tools, models, network reach, compute. A child starts from what its parent has left and reserves its budget from the parent. There is a property test: in 200 random sequences of spawn, spend and close, no session overspends and no tree exceeds what the root was granted. Depth is capped at 3.

5.5 Tokens, ArgoCD, audit

• Interceptor route POST /agent-installation-token, authenticated by TokenReview as Clearance’s service account. It enforces an independent outer bound (an interceptor-side repo allowlist; permissions limited to contents, pull_requests, check reads; never administration or workflows), so a Clearance bug cannot widen it.
• ArgoCD: p, role:clearance-lower, applications, sync, *-lower/*, allow. Tower’s role:tower-sync is */* today. Unverified: that the glob matches your <app>-lower AppProject names.
• Audit: hash-chained JSON lines, arguments stored only as a hash. The chain proves consistency, not completeness: truncating the tail leaves a valid chain, so checkpoint() must be anchored somewhere the writer cannot rewrite. Tests cover edit, delete, reorder, truncation and rewrite-after-anchor. Superseded design for the anchor (2026-10-28, section 0.1): each audit line becomes an evidence record; the Clearance ledger commits only their hashes to a Tessera log; checkpoints are countersigned by a witness Clearance does not operate, which never accepts a smaller tree, so even the key holder cannot truncate and re-sign unseen. clearance verify recomputes the hashes from the tenant’s own files and names an altered line.

5.6 Preflight

A case names deterministic verifiers, a path scope, and denial and token budgets. Two guards against a harness that always says yes: a run that changed nothing cannot pass, and every case must fail a synthetic unfixed run. Seed cases come from your own dead-ends list (liveness probe path, function-auto-ready, RabbitMQ 4.1 versus operator 2.23, hand-applied XRD reverted by selfHeal, Trivy-gated Java pins, the broken canary). Two are written (autopilot/preflight/cases/); the rest are listed in the diagram. Other agent types get their own suites, graded by deterministic checks such as schema-valid output and no forbidden tool calls.

6. Backend infrastructure

6.1 Inventory: what backs an agent, and who manages it

The model proxy’s routes are durable git (autopilot/config/model-routes.example.yaml): an alias maps to a backend. A run names an alias and never holds a key or a URL.

6.2 What Modelplane is

Read from its repo (b3b3f2f, 2026-09-25) and docs, not from memory. (modelplane.io refused my connection; the project is at modelplane.ai.)

• An open-source control plane for AI inference, v0.1, API modelplane.ai/v1alpha1, Apache 2.0, from Upbound. It requires Crossplane on a control cluster above a fleet of inference clusters.
• Built the way Hangar is: the entire system is Python composition functions with no custom controllers; the fleet scheduler is “a pure function of observed state” using CEL over typed hardware attributes; two personas share one API.
  • Platform team: InferenceClass (a hardware recipe), InferenceCluster (provisioned on EKS, GKE, AKS, Nebius or Vultr, or source: Existing), InferenceGateway (OpenAI and Anthropic APIs).
  • ML team: ModelDeployment, ModelService, ModelCache.
  • Composed: ModelReplica, ModelEndpoint.
• A ModelEndpoint can also point at an external provider, so one ModelService can front self-hosted replicas and a hosted provider with weighted traffic and failover.
• The gateway supports per-caller API keys (stamping identity on requests and usage records) and a “run behind another gateway” mode that trusts an x-modelplane-caller header.
• Requirements on an inference cluster: Kubernetes 1.34.2 or newer with DRA, NVIDIA drivers, a load balancer, a ReadWriteMany StorageClass for caches.
• AWS EKS is provisioned natively. OCI (OKE) is “planned”, so it works today only by bring-your-own with source: Existing.

6.3 Recommendation: adopt it, behind one contract, later

Building fleet scheduling, GPU cluster provisioning, weight caching and an inference gateway is exactly what Modelplane is. Hangar should not. But it is v0.1, so:

• The only thing Hangar depends on is an OpenAI or Anthropic compatible URL. Hosted providers and Modelplane are interchangeable behind the model proxy. Nothing in phases A to C needs Modelplane, and hosted models come first.
• Pin the version (git-source pinned tag, your standing rule), and treat its v1alpha1 API as able to change.
• A contract test in Preflight: a completion through the proxy against the inference gateway.

6.4 Where it collides with your principles

These are explicit decisions, not details:

1. It holds credentials for other clusters. The hub reconciles inference clusters, so it holds their credentials (and, for provisioning, broad cloud credentials; the GKE tutorial needs projectIamAdmin). That contradicts “no cluster holds another cluster’s credentials”. Containment: inference is its own fleet class with its own hub per environment (a dev hub never holds prod credentials), dedicated cloud accounts, and inference clusters that hold no app data, agent data or Hangar secrets. Prefer source: Existing clusters built by Terraform in a controlled account, so the hub holds only a kubeconfig, not cloud admin.
2. It owns each inference cluster and installs Envoy Gateway. It assumes exclusive ownership and owns the GatewayClass, so it cannot share a cluster with your Hangar clusters, which run Traefik (GatewayClass traefik, verified 2026-10-28). Inference clusters are dedicated: registry type inference, not dev or upper.
3. It needs its own Crossplane packages. You have been bitten by package dependency-lock corruption on a shared Crossplane. The hub is its own cluster (registry type hub), never the Airframe Crossplane.
4. source: Existing takes a kubeconfig Secret, a persisted credential, against never-persisted. Unverified: whether short-lived cloud auth (an exec plugin) works there. If not, this is an accepted, documented exception on a bounded fleet, delivered by External Secrets and never in git.

6.5 Managed the Hangar way

• Hub config (InferenceClass, InferenceCluster, InferenceGateway, ModelDeployment, ModelService) is durable and lives in git, in gitops-cluster-<hub> from an apron template of type hub, with a tenants repo for ML teams’ namespaced resources. Modelplane is installed by a pinned ArgoCD Application.
• Replicas and scaling are runtime, derived by Modelplane and KEDA. They are not committed.
• Cloud and provider keys come from Infisical through External Secrets. Modelplane’s own tutorial has you kubectl create secret; here it is an ExternalSecret.
• Budgets stay in the model proxy. Modelplane advertises no usage caps or token metering. Budgets, model allowlists and per-run caller identity are the proxy’s job. Modelplane’s usage records carry the caller, so setting x-modelplane-caller to the session id lets them be joined to task_id. Unverified: what those records actually contain.
• GPU metrics and Modelplane’s engine metrics feed the same Prometheus.

7. Failure modes

8. Open decisions

1. Commit signing for agent PRs. provenance verifies human gitsign against public Sigstore. Options: sign with the self-hosted Fulcio (workload trust root) and extend the gate, or exempt agent PRs. Decided (D1): the first, under a new Hangar root CA. It is also one of the three signals Clearance’s provenance detector needs for confirmed agent (section 0.1). Never mix the two trust roots.
2. AgentRun as an XR, a bare Job, or a controller. Recommend the XR. A Job alone loses status, policy and Tower visibility; a custom controller repeats a reflex you already decided against, and Modelplane is evidence it is unnecessary.
3. Standalone Clearance or a Backstage module. Decided (D3): standalone, federating Backstage’s MCP actions. Since 2026-10-28 it is a product with its own repo (D16). 3a. Where the frontend lives: a new Backstage plugin, or a Tower tab. Decided 2026-09-27: a new, standalone-installable Autopilot plugin beside Tower, not a tab inside it. Tower’s whole chrome (EnvPicker, AppPicker, the release matrix, promotion order) is built around one app’s environments and releases; an AgentRun does not live at that coordinate; it can be triggered by an event with no app in the picker, a scheduled task touching several repos, or a team fanning out across them. Forcing that into Tower’s app-scoped tabs would mean inventing a fake app context, or a second incompatible navigation model bolted onto Tower’s sidebar. The precedent already in this codebase: Glidepath is a standalone-installable CI/CD engine, and Tower’s Glidepath tab is a thin reuse of Glidepath’s own backend routes, not a reimplementation. Autopilot gets the same relationship: a fleet-wide console (every run, every agent, budgets, the breaker, Preflight results, audit) in its own plugin, and a narrow Tower “Agent” tab for the one case that is genuinely app-scoped and conversational (a session-type agent like gate-copilot chatting inside the app a human is already looking at), backed by the Autopilot plugin’s own API, exactly like the Glidepath tab is backed by Glidepath’s.
4. A separate GitHub App for agents, so their rate-limit bucket is their own. Decided (D4), reconfirmed 2026-10-28. It is not the Clearance provenance App (section 0.1).
5. Audit store. MinIO object lock must be enabled when the bucket is created. Revised 2026-10-28: integrity comes from the witnessed ledger; object lock is retention only (D5, section 0.1).
6. Runtime boundary: Clearance as an AuthZEN PDP behind Traefik Hub, or its own gateway. Open (Clearance architecture ch. 15). Recommendation: PDP, keeping the own gateway as the fallback. Hangar runs Traefik Proxy, not Hub, so the own gateway is what ships now.
7. Hub isolation and credentials (section 6.4). Decide before any hub exists.
8. When to trial Modelplane. After phase D, once agents run and their spend is bounded, and on a dedicated kind cluster first (its getting-started guide targets local kind, but it needs a cloud account for GPUs).
9. Debugging without exec. Accept it, or build a recorded, time-boxed, approved break-glass as its own initiative, never a normal backlog item (your policy).

9. Built versus not

As of 2026-11-02: clearance v0.26.1 (406 tests, Postgres included) and autopilot gateway-v0.3.27 (554 tests), both with no cluster, and the whole runtime boundary live on hangar-dev. Section 4 has the per-component table.

Not built: the tools without a backend (chat.*, app.api.get, xr.request; airframe.validate in the image), airframe.apply, Backstage MCP federation, stable agent identity by ID-JAG (D19), a durable event cursor, the deployment gate’s admission side, evidence signing, checkpoints and a witness, and anything on hangar-prod beyond Tower reading clearance-api. The 2026-10-28 text of this section (308 tests, no real adapters, drafts never applied) is superseded.

10. What I did not verify

• Whether the Backstage mcpActions endpoint is live on any cluster.
• Whether HolmesGPT can call an MCP server authenticated by a service-account token, and forward a task id.
• Whether Tower’s Tier 1 write actions were live-verified after the 2026-09-15 handoff.
• The provenance gate’s internals and where the per-app signer allowlist lives.
• The *-lower/* ArgoCD glob against your AppProject names.
• Time-driven re-invocation through the function response TTL on your Crossplane version.
• Whether provider-kubernetes Objects for Namespace, ResourceQuota, NetworkPolicy and Job apply cleanly with the grants drafted.
• pgvector in the CloudNativePG image.
• The upstream Kubernetes Agent Sandbox and GKE Agent Substrate: known only from a third-party survey, not read at source (U11).
• Modelplane’s usage-record contents, short-lived-credential support for Existing clusters, and behaviour on your Kubernetes versions (prod is 1.37; the dev cluster’s version I did not check).

Sources

• Modelplane, the repository, and Building Modelplane on Crossplane.
• Modelplane docs read from the repository: overview/how-it-works, platform/inference-cluster, platform/inference-gateway, platform/providers, models/model-endpoint, getting-started/build-the-platform.