Docs · Glidepath

Release state

Every release to an upper-cluster environment has a record on the dev cluster that says where the release is: proposed, merged, rolling out, healthy, failed, superseded. The record is what turns a stream of low-level facts from the prod cluster (a Rollout changed phase, a sync failed) into the release events the rest of Glidepath consumes, and it is what lets the platform tell a release from a scale, a restart or a manual edit.

This page is the reference. The decision and its history are in ADR-0021 (with the phase 0 findings and phase 1 results); the event vocabulary is ADR-0022.

The record

One ConfigMap per release, release-tracking-<chain-id>, in the Application’s own namespace (app-<name>-cicd) on the dev cluster. Labels: hangar.io/subcomponent=release-tracking, hangar.io/app, hangar.io/env, hangar.io/cluster. The release is identified by releaseId, <chain-id>:<cluster>/<env>, which also rides on the Rollout as the hangar.io/release-id annotation so a fact can be joined to its record.

The record is not deleted when its outcome is read (it used to be). The state machine needs it afterwards, and the sweeper removes it past its retention.

Outside Glidepath, Tower’s backend reads the records (list with the labels above, as its read identity on the dev cluster) for three things: the releases a rollback may target, the images an image prune must keep, and each release record’s outcome (clean, rolled back, replaced before healthy, stopped before production), from state, healthyAt, supersededBy, rolledBackBy, rollbackReason and image. Tower writes the outcome into the release record in gitops-<app> once it is final, so it outlives the sweep; a release whose records were swept first, or written before phase 4 (no image), has no outcome. See Tower’s docs/release-records/index.md.

The state machine

Tower’s Promote to a Flight environment runs the same release Pipeline, from the promote-release TriggerTemplate glidepath-app renders into each Kubernetes app’s -cicd namespace; Backstage fills its params and creates the PipelineRun. Until 2026-10-08 the Backstage backend opened the release PR itself and wrote no record, so the relay dropped every fact for a Tower-promoted release as no-record.

sync-failed is not terminal. healthy, aborted, degraded, superseded, rolled-back and closed are.

What a fact does

Facts are at-most-once and arrive repeated, out of order, and describing things that are not releases (all of it observed on a real release). The reducer (reducer.go, a pure function, so every row below is a plain test) handles it like this:

The pod template hash is deterministic for a given template, but it is not an identity for “the same image”: rolling back to an earlier image can produce a different hash. The release-id, not the hash, identifies a release.

How a fact is applied

• Events go out before state is written. If the write then fails, the next fact repeats the same deterministic event ids (derived from the release-id, event type and outcome), which the broker’s Triggers collapse. The other order could lose an event for good.
• A failed forward returns 502 and writes nothing. The sender does not retry, but the Rollouts engine re-sends current state every 15 minutes (its heartbeat), so the retry is the next heartbeat. Rollouts notifications are at-most-once and fail silently (phase 0); the heartbeat, and the prune CronJob that resets the engine’s own state, are what make the stream recover.
• A write conflict re-reads and re-reduces. There are two relay replicas.
• A record the relay cannot write (no Role yet, or it vanished) degrades to the stateless behaviour of phase 1: the deterministic ids still make repeats harmless.
• The hook Jobs are gone (phase 3b/3c). While both ran, emit mode dropped the hook Jobs’ copy of an event when the facts owned the release, and forwarded it as a fallback when they had not; that code, the /outcome endpoint and the hook script were deleted with the Jobs. A release whose facts are lost now waits for the next heartbeat (15 minutes), and the sweeper raises ReleaseStalled if none arrives.
• Shadow and emit keep separate sent-sets (emittedShadow, emittedLive). In shadow (opt-in; outcomeRelay.factsMode defaults to emit) the relay logs the event it would send as shadow-event and forwards nothing. Flipping to emit then sends the history instead of believing shadow already did.

The two sources

Tenant Applications retry (limit 5, backoff up to 10 minutes), about 15 minutes to a final Failed, and operationState.phase stays Running through the retries. A trigger on phase == Failed alone would be 15 minutes late, which is why the first failed attempt fires.

Stalls and retention

release-record-sweeper (control plane, every 10 minutes):

• raises a ReleaseStalled Event on a record that is merged with no fact for 45 minutes (the sync never started, or facts are not arriving), or progressing with no fact for 45 minutes (the heartbeat arrives every 15), once per state;
• deletes records past 14 days: terminal and closed records by stateAt, proposed and state-less legacy records by creation time. A merged or progressing record is never swept, and neither are the newest five records per app, environment and cluster that reached healthy (the rollback-eligibility window, below).

The knobs are env vars on the CronJob (RELEASE_STALL_MINUTES, RECORD_TTL_DAYS, KEEP_HEALTHY).

Rollback (ADR-0021 phase 4)

A rollback is a release of an earlier image: Tower’s Roll back starts the app’s rollback-release TriggerTemplate (glidepath-app), the same release Pipeline as Promote with release-kind: rollback, rollback-of and rollback-reason, and config-revision so cicd.yaml is read at the app repo’s main while provenance still checks the image’s own commit. open-release-pr records the kind, opens branch release-rollback-<env>-<sha8> and titles the PR Rollback:.

The content gates (sast, image-scan, sbom) work out for themselves whether the promoted image is an eligible rollback target: extract-promoted-image’s rollback-eligibility step reads this app/environment/cluster’s records and answers yes when one of the newest five that reached healthy ran the same image. Then the gate runs its check as <task>-advisory with onError: continue: the check run passes and the PR comment shows what it found. Integrity gates (provenance, values, commit signing) and approvals are unchanged. The PR’s own text and trailers are never trusted for this.

Permissions

Per Application namespace (glidepath-app, identity/release-record-relay.yaml), not cluster-wide:

The relay’s cluster-wide get on ConfigMaps from phase 1 (glidepath-relay-records) remains until every namespace has the Role, then goes. The two infra apps (skyport-auth, skyport-broker) have no release flow and no Role; the relay and sweeper skip them.

Operating it

# every release's state, newest last
kubectl get cm -A -l hangar.io/subcomponent=release-tracking \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,STATE:.data.state,SINCE:.data.stateAt,LAST_FACT:.data.lastFactAt

# one release in full
kubectl -n app-gate-api-cicd get cm release-tracking-<chain-id> -o yaml

# drift and stalls
kubectl get events -A --field-selector reason=ReleaseDrift
kubectl get events -A --field-selector reason=ReleaseStalled

# what the relay would send, if factsMode is shadow
kubectl -n platform-system logs -l app=glidepath-relay --prefix | grep shadow-event

Tests

• glidepath/broker: go test ./cmd/glidepath-relay. The reducer’s rows above, state persistence, a failed forward, a write conflict, an unwritable record, drift, supersede, shadow-then-emit, and the Argo CD endpoint.
• charts/glidepath-catalog/tests/mark_release_merged_test.sh, charts/glidepath-catalog/tests/rollback_eligibility_test.sh and charts/glidepath-control-plane/tests/release_record_sweeper_test.sh run the real Task and sweeper scripts against a stub kubectl.

Not built yet

Moving the events onto the spec’s vocabulary (ADR-0022). Recording on the release that a Tower Promote full skipped analysis (Backstage audits and announces it; the fact carries no such field yet).

A PR closed without merging

The merge signal is the push to main, and an unmerged close produces no push, so mark-release-merged never sees one. Instead, when open-release-pr opens a new release PR, it closes out the earlier proposed records of the same app, environment and cluster (close_stale_proposals): each whose PR GitHub reports closed and unmerged becomes closed. A record still open, merged, or unreadable is left alone, and this never fails the release. That covers the common case, a release replaced by a newer one (2026-10-30: five such records made Tower’s Ops Wall show releases awaiting approval whose PRs were long closed). A PR closed with no later release of that environment still stays proposed until retention deletes it (14 days).

Live verification (2026-10-06)

Every scenario above was exercised against the real clusters, not only in unit tests: real gate-api releases on prod, and a synthetic Application namespace on dev driven through the relay’s real endpoints (with the real mark-release-merged Task and the real sweeper CronJob).

Not exercised live: a forward failure and the 502 retry (needs emit mode, unit-tested), a write conflict between the two relay replicas (unit-tested), and a PR closed unmerged with no later release (the gap above; close_stale_proposals is covered by its script test). The retention of proposed and state-less legacy records is covered by the sweeper’s script test, because a record’s creation time cannot be back-dated.