An evaluation of Hangar as the company's next-generation application and AI agent platform: its architecture, goals, roadmap and current state, with a decision and the conditions attached to it.
Hangar's design principles are the strongest I have seen in a proposal of this kind, and I want the company's platform built on them. The implementation, however, has been proven on two laptop-hosted clusters under a personal GitHub account by one engineer, and the roadmap drives toward an AI-operability target without a matching track for production hardening. I am approving a funded pilot, with five gates that must pass before any customer-facing workload moves onto it.
Hangar is an internal developer platform built from six products. Airframe is the service catalog: Crossplane XRDs and compositions plus one Helm chart (airframe-application) that every environment renders through. Glidepath is CI/CD on Tekton and Pipelines-as-Code, driven by a single developer-facing cicd.yaml. Tower is a Backstage plugin for releases, deployments and configuration. Apron is a template that stamps out per-cluster GitOps repos. Autopilot, added on 26 Sep, runs AI agent workloads through a policy gateway called Clearance.
One rule holds it together: every durable change is a reviewed git commit, no cluster holds credentials for another, and outcomes flow back as CDEvents rather than API calls. Autopilot extends this with a second plane: ephemeral agent runs are Crossplane claims with hard deadlines that can only narrow what git grants.
| Area | Rating | Summary |
|---|---|---|
| Architectural principles | Strong | Git as the only write path, no cross-cluster credentials, delegated over interactive access. Consistent and applied everywhere. |
| CI/CD (Glidepath) | Strong | 17 ADRs, keyless signing, provenance, OTel traces per flow, loud governance stubs, a published known-gaps list. |
| Service catalog (Airframe) | Adequate | 13 XRDs, live-verified. Relies on Crossplane writing files to GitHub, which has caused rate-limit exhaustion and a data-loss incident. |
| Developer portal (Tower) | Adequate | Rich UI, but about 25k lines with no tests, distributed by copy-paste, and write routes that authenticate without authorizing. |
| AI agent platform (Autopilot) | Promising | Excellent threat model and policy core. Real adapters, auth and the runtime are still proposals. |
| Multi-cluster and topology | Weak | Prod depends on the dev cluster for secrets and bootstrap. Cross-cluster paths use NodePorts, SSH tunnels and host IPs. |
| Identity and access | Weak | Personal GitHub account with a classic PAT holding delete_repo. One hardcoded admin in Backstage. No enterprise SSO. |
| Day-2 operations | Gap | No backup and restore design, no platform SLOs, no upgrade playbooks. Prod observability is scaled to zero. |
| Documentation | Adequate | Deep and candid, but written as running logs (the catalog design is 155 KB). Stale names and some inconsistent figures. |
| Delivery risk | High | One engineer across 12 repos, 28-week plan. The roadmap lists this as its first risk. |
Upper and lower environments get separate ArgoCD AppProjects. The rejection was attack-tested live with throwaway Applications, not asserted from YAML.
cicd.yaml has a strict JSON Schema and is scaffolded at onboarding (ADR-0017). Nobody writes Tekton YAML.
ArgoCD's valuesObject precedence over valueFiles was tested with a planted hostile override. It is used as a safety mechanism on purpose.
Every claim is labelled Built, Draft, Proposal or Unverified. Stub gates report "stub", never "pass". Known gaps are published with evidence.
Two planes, narrow-only budgets, tripwire tools, expiry that needs no gateway, hash-chained audit with an external anchor, and evaluations that must fail a seeded bad run.
The docs catch and reverse the reflex to write controllers: provider-sql, the RabbitMQ operators, CloudNativePG and the upstream Agent Sandbox were chosen over bespoke code.
Ordered by severity. Each carries the evidence I checked. "High" means it blocks production approval.
Bootstrap-tier Crossplane runs only on the dev cluster, and so does the only Infisical instance. The prod cluster's secret stores pull from it at http://dev.kiac.local:31800 over plain HTTP, through a NodePort that works because both clusters share one home network. A dev-cluster outage or compromise therefore reaches prod secrets and blocks new environments everywhere. The org-level INFISICAL_ADMIN_TOKEN already has blast radius on both clusters, which the design accepts.
Evidencekind-prod-infisical-migration-plan.md lines 40, 156 and 201; service-catalog-design.md §0 and item 8 "Still open".
GitHub Apps cannot create repos under a user account, so provider-github runs on a classic PAT with repo and delete_repo. Decommissioning an app deletes both of its repos. Branch rulesets on private repos need a paid plan, and the API refuses to show them on this account, so the guardrails that make release checks required cannot be audited.
Evidenceservice-catalog-design.md §1; Glidepath known-gaps.md #14; airframe/docs/user/decommission-app.md.
The backend routes for ArgoCD sync, promote, config edits and cicd.yaml edits read the caller's identity only to log it. The permission policy covers catalog and scaffolder actions and a single hardcoded admin (user:default/jfillman), so any signed-in user can sync any mapped app on any cluster. The ArgoCD account behind it is described as role:tower-sync */*. This contradicts the lower-only write policy the Autopilot design relies on.
Evidencebackstage/packages/backend/src/glidepathProvenance.ts lines 1587 to 1640; permissionPolicy.ts; autopilot/design.md §5.5.
I found no backup or restore plan for cluster state, CloudNativePG databases, RabbitMQ, the audit bucket or the Crossplane-managed GitHub state. Infisical's root encryption key is noted as "unrecoverable if lost" with no escrow procedure. Prod observability is scaled to zero for capacity, so prod has no monitoring today.
Evidencecluster-provisioning.md line 144; roadmap.md §6 risks; search across all 12 repos for backup, Velero and restore.
Bootstrap XRDs reconcile RepositoryFile objects through GitHub's Contents API. That produced an annotation flip burning about 280 calls a minute, a retry loop burning about 16,000 calls an hour against a 5,000 budget shared with Backstage, a lost external-name that turned into repeated create attempts, and a Usage/finalizer deadlock that is "suspected resolved, not proven". It also needs a patched fork of provider-upjet-github. The team has hardened it well, but a reconcile loop is the wrong tool for a one-time scaffold.
EvidenceAirframe README.md status notes of 23 and 24 Sep; service-catalog-design.md §0 and the 18 Aug incident; fork commits ea893ce and 7530c3d.
An unsigned release PR failed provenance and was merged anyway, so commit signing is not enforced in practice. Signatures are checked at release time, but nothing verifies them at admission, because policy-validation is still a stub and Kyverno carries only Testkube policies. The dev cluster's CNI does not enforce NetworkPolicy, and ground env files are committed straight to main with no validation.
Evidenceservice-catalog-design.md 25 Sep round; Glidepath release-guardrails.md, known gaps #4 and #5; apron/30-policy.
M0 to M5 measure progress with the Airframe scorecard, which grades how operable Airframe is by agents. That is a good metric, but it is the only one. SSO, team tenancy, a cloud target, HA, DR, platform SLOs and upgrade testing appear in no milestone. The "parachute" acceptance test proves an agent can provision an app, not that the company can run one.
Evidenceroadmap.md §2 milestones and §6 risks; airframe-ai-friendly.md §2.
Each cluster runs two ArgoCD instances, Crossplane with four or more providers, Tekton, PaC, the CDEvents broker, ESO, Infisical, Kyverno, Sloth, Argo Rollouts, Contour, a full observability stack and HolmesGPT. Each app gets a source repo, a gitops repo and an entry in each cluster's tenants repo. The shared idp-cluster-baseline chart meant to keep clusters thin was never built, so every cluster is a full vendored copy of the template. Custom code (the broker interceptor, dora-exporter, results relay, rollout-watcher, provider-infisical, Tower, Clearance) all needs owners.
Evidencegitops-strategy.md §1 to §3; apron/README.md "What this template does not model yet".
Crossplane on this installation re-renders compositions only every 60 seconds (U2), which is fine for batch tasks but slow for interactive sessions, and the team already concluded that hard expiry must come from the Sandbox controller. Clearance keeps sessions in memory on a single replica. Six workload shapes, a team orchestrator and a self-hosted inference hub are a lot of surface before the first real adapter exists.
Evidenceroadmap.md §5 answers U2 and U11; design.md §7 failure modes.
The Autopilot docs claim 255 tests; I ran the suite and got 218 passed, 4 skipped. The Autopilot README says 15 CEL rules while the design says 19 (the code has R000 to R019). The scorecard defaults to /Users/jerf/tech and needs Helm, so nobody else can reproduce the published 27 and 41 scores. Airframe and Apron still link to jfillman/idp and platform-cicd. Design docs are session logs rather than as-built references.
Evidencepytest -q in autopilot on 27 Sep; autopilot/README.md; tools/airframe-scorecard/scorecard.py.
type: mgmt cluster with its own change bar. Dev goes back to being disposable.releaseGuardrails.verifyImages or the Sigstore policy controller, so an unsigned image cannot run even if a gate is bypassed.idp-cluster-baseline before a third cluster exists, as §8 of the GitOps strategy intended.SandboxClaim the run primitive, keep Crossplane for durable agent definitions, and give Clearance a persistent session store and two replicas.Production approval follows when these pass. Each is a demonstration, recorded the way the project already records live verification.
Topology. Prod keeps serving and syncing with the dev cluster powered off, and no prod component resolves a dev hostname.
Closes F1
Identity. The platform runs under a GitHub organization with no classic PAT, rulesets are applied from code, and users sign in through corporate SSO.
Closes F2
Authorization. A non-owner is refused on every Tower write route, and nobody can sync an upper environment from the portal without an explicit grant.
Closes F3
Recovery. A timed restore drill rebuilds a cluster, a CloudNativePG database and Infisical from backup, with the recovery time written down.
Closes F4
Supply chain. An unsigned image is refused at admission on an upper cluster, and a release PR with a failing provenance check cannot be merged.
Closes F6
Management tier, GitHub org and SSO, Tower authorization, DR, admission policy. Continue Airframe M1 (contract and strict schema) in parallel, since it has no conflict.
Onboard two real non-critical services on the cloud target. Measure onboarding time, change failure rate and support load, and staff at least three more engineers.
Real adapters, task and event agents, Preflight as a CI gate. Agents stay dev-only until the audit anchor and cost accounting are live.