Hangar

Docs · Glidepath

ADR-0029: Ground and Flight registries; Flight images are promoted by digest before the release merges

Status: Accepted (2026-10-27). Supersedes decision 3 of ADR-0025. Phase 0 (spike) done; phase 1 (configuration and digest pinning) live on hangar-dev 2026-10-27; phase 2 (the promoter: Pipeline, StepAction, broker /promoter-token) live-verified on hangar-dev 2026-10-28; phase 3 (the fan-in, the image-promotion registry entry and PR comment) live-verified on gitops-gate-api#57 2026-10-29; phase 4 (upper-cluster admission, Apron admission.flight) in Audit 2026-10-29, pullFromFlight and enforce (Deny, fail closed) on hangar-prod 2026-10-30, after every app’s prod release ran from Flight (backstage, boarding-api, flight-api, gate-api); phase 5 (Tower: Promoted step, Flight/Ground chips, Release Record fact) live 2026-10-29 (tower v0.104.0; v0.105.0 runs in the lab); phase 6: cloud targets in a Flight environment refused while promotion is on (PROM-10, live 2026-10-30) and Flight retention built (promotion.retention, off; the prune workflow is in the promoter repository, not run yet). Detail: registries.md.

Context

Every image lives in one registry (ghcr.io/<owner>): builds push there and every cluster pulls from there, so “released” and “built” are the same place and an upper cluster can run anything a build ever pushed. ADR-0025 retired the image-promotion stub because, with one registry, it had nothing to do. A second registry that only a promoter can write gives the check a real job: an upper cluster can only run an image that went through the gates and was copied.

Decisions

  1. Two registries. Ground: written by builds, read by lower clusters, Testkube and the gates. Flight: written only by the promoter, read by upper clusters with a read-only credential. In the lab, Flight is ghcr.io/riley-roooo, a separate GitHub account. Configuration is admin-owned (catalog registries, promotion), never cicd.yaml.
  2. Release files pin a digest and record the source. release.image.digest (what runs; airframe renders repository:tag@digest) and release.image.source (the Ground repository, where the evidence was verified). A Flight environment’s release file pins the Flight repository from the moment the release PR is opened.
  3. Promotion happens before the merge, when every required gate is green on the release PR’s head commit (an event fan-in, not a polling pod). Merging before promotion fails closed. As built (phase 3), the fan-in is a step in every gate’s finally with a gate ledger in the release record, not the relay’s reducer: the relay consumes no broker events and runs only where remote clusters exist, and the gate’s own finally already knows its verdict.
  4. The copy keeps every digest. Index, platform manifests, the legacy .sig/.att tags and the OCI 1.1 referrers are copied by digest; nothing is re-signed, so Rekor entries and their integrated times stay valid.
  5. The copy runs in GitHub Actions. GHCR does not accept GitHub App installation tokens, even with packages: write and the package linked to the installed repository (phase 0: push and read both refused). The promote pipeline uses the hangar-flight-promoter App only to start the promote workflow in riley-roooo/flight-promoter; that run’s GITHUB_TOKEN is the only credential that writes Flight. Ground is private and GitHub will not grant another account’s repository access to a package, so the workflow reads Ground with a read:packages token stored as an Actions secret (persisted, read-only).
  6. Upper-cluster admission verifies the promotion attestation, not the Chains provenance. The workflow signs https://hangar.io/attestations/promotion/v1 keyless on public Sigstore with its own workflow identity; cosign 3 embeds the Rekor inclusion proof, so Kyverno verifies it offline. Chains’ attestations carry no inclusion proof and Fulcio and Rekor run only on hangar-dev, so an upper cluster cannot verify those. The gates verify the provenance before the promotion that this attestation records. The “Flight images only” rule is a separate plain ValidatingPolicy: an ImageValidatingPolicy fetches every image it matches with its own credential.

Consequences

  • The promotion attestation is in the public Rekor log: the Flight repository, digest, source and run URL are public; the image is not.
  • Two attestation storage schemes coexist per digest; the copy covers both.
  • airframe-application before v0.3.166 rejects digest/source, so promotion.pinDigest stayed off until every cluster’s chart pin was at v0.3.166 or later (on in the lab since 2026-10-27). Tower (tower#143) and Backstage (backstage#198) must parse repo:tag@digest before then too.

Phase 0 evidence (2026-10-27)

boarding-api 2.0.5-832e475 copied to Flight by the workflow: every digest equal, both platforms, the same five attestations discoverable through the Flight reference; the promotion attestation verified offline against the workflow identity, a wrong identity rejected; Kyverno 1.19 on hangar-prod (Audit+Warn, scratch namespace) admitted the promoted image and warned on the Ground copy and on an unpromoted Flight image.

Open after phase 6

  • Cloud promotion: a cloud Flight environment is refused (PROM-10) rather than promoted; release pin PRs would need the fan-in and the cloud deploy would copy from Flight. No app has one today.
  • Flight retention is off until promotion.retention.enabled is turned on and a dry run checks it.
  • The promote workflow trusts whoever can start it: the attestation proves the copy went through the workflow, not that the gates were green. Glidepath starts it only after PROM-03.
  • Admission fails closed, so a Kyverno outage on an upper cluster refuses new tenant pods.