Docs · Glidepath
Glidepath: known gaps
Found in use on 2026-09-26/27 while building baggage-api and the values-validation gate. Each has a reproduction or
evidence, so they can be picked up later without re-deriving them. Roughly ordered by how much they cost.
| # | Gap | Evidence | Direction |
|---|---|---|---|
| 1 | Image version has no Python support. build-image takes the version from a git tag, then package.json, then pom.xml, else 0.0.0. A Python app is always 0.0.0-<sha>. |
baggage-api images are 0.0.0-893e163; boarding-api is 1.1.0-8f9acd9. |
Fixed and live-verified 2026-09-27. jfillman/glidepath#2 (build-image reads pyproject.toml’s [project] version via yq -p toml, falls back to a plain VERSION file) and jfillman/airframe#5 (pythonapplication’s pip branch scaffolds a VERSION file) both merged; airframe tagged v0.3.94 and the pin bumped in apron/glidepath/gitops-cluster-dev, synced on dev. Composition reconciled, committed baggage-api’s VERSION (0.1.0) straight to main, a real build ran, and the resulting image-ref is ghcr.io/jfillman/baggage-api:0.1.0-ae82839 - deployed to dev. boarding-api/flight-api will pick up the same fix on their next resync. |
| 2 | The onboarding resync can report “up to date” when it is not. For boarding-api it said “gitops repo: .tekton/ already up to date, nothing to deliver” while the gitops repo lacked the new pull-request-values.yaml; a second run delivered it. Likely it compared against a template ConfigMap that had not refreshed yet. |
onboarding-resync-q8m67 log, 2026-09-27 00:35. |
Compare the delivered set against the rendered template set by file name and hash, not by the previous run’s state; retry once when the template ConfigMap changed during the run. |
| 3 | Deploy paths are hardcoded to platform/envs/<env>.yaml. A standalone Glidepath would write an Airframe-shaped file nothing reads. |
deploy-manifests, open-release-pr, the PaC exemption, ephemeral-envs, Tower. |
deploy.releaseFile in cicd.yaml (design in Hangar docs/autopilot/release-file-split.md). |
| 4 | An unsigned commit fails the provenance gate on an upper release, and so image-promotion. Agent commits are unsigned. |
baggage-api release PR #2: commit has no gpgsig. |
Decision D1 (self-hosted Fulcio for agent commits). Meanwhile a clearer message: which commit, and how to sign. Partly addressed 2026-10-20 (ADR-0027): an agent’s PR merged on GitHub by an allowed person passes in merger mode when the admin turns the GitHub-merge path on for the app; the agent’s own commits stay unsigned, so strict still fails them. |
| 5 | Config-only pushes are not validated. Pushes touching only platform/ or cicd.yaml never build, so a human’s direct edit of an env file on main is checked by nothing. |
The exemption in deliver-onboarding-files. |
A push-triggered validate-only flow for env files, or a required PR path for them. |
| 6 | Adding a guardrail can hang releases. image-promotion waits for every gate in releaseGuardrails; a repo without the gate’s .tekton file never reports it. |
wait-for-release-guardrails. |
Register a gate only after every gitops repo has the file (documented), or have the wait skip gates a repo does not carry. |
| 7 | Pull-request pipelines need a preview label and do not run on opened. A PR shows no checks until labelled. |
baggage-api PR #2 had no checks until the label was added. |
Say so in the PR comment the platform posts, or run the build on opened when the label is already present. |
| 8 | The webhook path is fragile. PaC deliveries reach the dev cluster through a temporary tunnel and a kubectl port-forward; when the forward died, pushes triggered nothing for about 19 hours with no alert. |
Restarted port-forward 8060:8080 fixed it. |
A named tunnel, and an alert on “no PaC event in N hours”. |
| 9 | A namespace can end up with no PipelineRuns at all, so Tower’s Pipelines tab is empty and nothing explains why. | baggage-api, before the U4 test. |
Keep the last N runs per flow regardless of the pruner, or show “no runs kept” in Tower. |
| 10 | Tekton steps run as non-root but scripts assume they can chmod a mount root. validate-values failed with chmod: changing permissions of '/work': Operation not permitted. Fixed 2026-09-27. |
TaskRun log. | Lint Task scripts for chmod/chown on volume roots. |
| 11 | The values-validation gate depends on the Airframe image tag. valuesValidatorImage must be moved by hand when the chart pin moves. |
glidepath-catalog/values.yaml. |
Derive it from the airframe-application chartVersion already pinned in glidepath-app. |
| 12 | open-release-pr writes releaseTracking.relayHostAliasIP from the dev node’s status.hostIP, an address a second cluster cannot route to; the outcome hooks then time out and a new Flight env’s first sync fails. |
baggage-api-staging, 2026-09-27; same as the parked bug in the Skyport handoff. Also listed in Airframe’s known gaps. |
Fixed 2026-09-28: the cluster registry entry takes an optional relayHostAliasIP (glidepath-control-plane clusters[]), and open-release-pr uses it when set, status.hostIP otherwise. The prod cluster’s entry sets 192.168.1.78. Found again on gate-api and checkin-api staging, after being fixed by hand several times. Obsolete since 2026-10-06: the hook Jobs, relayHostAliasIP and the identity chart no longer exist (ADR-0021 phase 3). |
| 13 | The values-validation gate hit three Tekton-script bugs on its first real runs (chmod on the mount root; workingDir on a read-only mount; an unpublished validator image tag). All fixed 2026-09-27. |
TaskRun logs. | Run a new Task once against a real PR before relying on it; add a fixture-driven test for Task scripts. |
| 14 | Documented 2026-09-27 (see github-ruleset.md); the problems it found are open. Was: the GitHub ruleset that enforces every guardrail was not documented. It is the thing that makes the checks required, so it is a mandatory install item, yet the docs describe classic branch protection “not IaC-managed” and say nothing of rulesets: scope, required check names (Pipelines as Code CI / <gate>-), bypass actors, which repos. releaseGuardrails and the ruleset’s required list must be kept in sync by hand; a mismatch either never blocks or blocks forever. The GitHub API refuses to show rulesets and branch protection for these private repos to a plain token (“Upgrade to GitHub Pro or make this repository public”), so an export is needed to audit it. |
docs/admin/release-guardrails.md “Branch protection”; gh api repos/.../rulesets returns 403 on the free plan. |
Export the ruleset JSON, audit it, document it, and make applying it an install step (ideally generated from releaseGuardrails, e.g. a RepositoryRuleset from the same list). Note the plan question: rulesets on private repos need a paid plan. |
| 15 | The required checks assume every PR is a release PR. All gates run only on release-* heads, so onboarding-resync, Tower Config, hand-edit and seed PRs would never satisfy the ruleset’s required checks. |
Audit of the exported ruleset against every gate’s on-cel-expression. |
Run every gate on every PR and pass immediately for non-release PRs, or add an admin bypass. |
| 16 | Bots write directly to the gitops default branch (the composition’s RepositoryFiles), which a pull-request rule would refuse. |
values-yaml.yaml in the ApplicationEnvironment composition. |
Make the writing GitHub App a bypass actor (--bypass-app-id). |
| 17 | platform/pr-env.yaml is never scaffolded onto an onboarded app repo, even though ephemeral-envs.yaml‘s ApplicationSet has always required it as its second Helm source. Enabling ephemeralEnvironments.pullRequest.enabled: true alone is enough to synthesize a working pr-build flow and ApplicationSet trigger, but the resulting preview Application sits in ComparisonError (failed to execute helm template command ... no such file or directory) forever, since nothing ever creates the file. Found live 2026-09-27: baggage-api-pr-2 (on dev) never provisioned; boarding-api and flight-api were missing the same file for the same reason. Fixed the 3 known-affected app repos directly (comment-only file, since all 3 apps’ container ports already match airframe-application’s rollout.ports chart default): jfillman/baggage-api#3, jfillman/boarding-api#8, jfillman/flight-api#8. |
baggage-api-pr-2 Application status, dev, app-baggage-api-cicd namespace. Root-cause fix merged and live 2026-09-27. jfillman/glidepath#3 merged; the live deliver-onboarding-files Task on dev (platform-catalog namespace, synced at fbb99b5) carries the scaffold step, confirmed by reading its rendered .spec.steps[].script directly off the cluster. It scaffolds platform/pr-env.yaml once via heredoc, guarded by [[ ! -f platform/pr-env.yaml ]], so a file a developer later edits is never force-overwritten - not re-exercised against a fresh live onboarding-resync run this session (no open PR to trigger one against), but the guard is unconditional and was read directly off the live Task, not just the source. |
|
| 18 | Release gates ignored release.yaml. With deploy.releaseFile (D9/AF-5) the release lands in a sibling <cluster>/<env>/release.yaml, but extract-promoted-image and verify-image-provenance only diffed */values.yaml, so image-scan, sast, sbom and provenance failed with “no */values.yaml changed” on every new app’s first release. |
gitops-gate-api release PR #4, 2026-09-27. |
Fixed 2026-09-27: both Tasks prefer */release.yaml and fall back to */values.yaml. |
| 19 | image-promotion waited forever on release PRs. It waits for every gate in releaseGuardrails, including values, but the values gate only triggered when a values.yaml changed. With deploy.releaseFile a release PR only changes release.yaml, so values never ran and promotion hung. |
gitops-gate-api PR #7, 2026-09-28. |
Fixed 2026-09-28: the values gate also triggers on release.yaml and passes with a note when no values file changed. Existing gitops repos pick it up through their next onboarding-resync PR. |
| 20 | The /retest <gate> hint in gate comments doesn’t re-run anything. PaC answers “cannot find the targeted pipelinerun sast in this repository”: the gates’ on-cel-expression only matches release-* pull_request events, not the retest-comment event. |
gitops-gate-api PR #4, 2026-09-27; PaC controller log. |
Match the retest-comment event in each gate’s CEL, or change the hint to closing and reopening the PR, which does re-run them. |
| 21 | Tower shows a crash-looping rollout as “Progressing” with no diagnosis. The Deployments tab matched no known issue and listed the pods as running while they were in CrashLoopBackOff (a readiness probe on / returning 404). |
checkin-api dev, 2026-09-28. |
Surface container restart counts and the last probe failure in Tower’s diagnosis rules. |
| 22 | A new app’s release sync hung at PreSync. The release hooks run as platform-outcome-hook, which airframe-identity only renders when releaseTracking is set, and the tenant-identity ApplicationSet only read values.yaml. With deploy.releaseFile, releaseTracking is in release.yaml, so the ServiceAccount never existed and the hook Jobs failed to create pods. |
gate-api staging, 2026-09-28: error looking up service account app-gate-api-staging/platform-outcome-hook. |
Fixed 2026-09-28: tenant-identity reads release.yaml too (ignoreMissingValueFiles), in the prod cluster’s repo and the Apron template. Obsolete since 2026-10-06: the hook Jobs, relayHostAliasIP and the identity chart no longer exist (ADR-0021 phase 3). |
| 23 | Built images carry no org.opencontainers.image.* labels. build-image’s kaniko invocations set no --label at all, so there’s no way to answer “what’s running in prod” from the image itself (crane config/docker inspect) without cross-referencing gitops release files - Tower’s own release-tracking regression (glidepathProvenance.ts hardcoding the pre-D9 path, see the Hangar autopilot handoff) is a direct symptom of that cross-referencing being the only path. |
Confirmed via direct read of build-image.yaml’s kaniko script: blocks - no --label flag present before this fix. |
Fixed 2026-09-29: build-image.yaml now sets org.opencontainers.image.{version,revision,source,created} via kaniko --label on both arch legs (applies unconditionally, no app-side Containerfile opt-in needed). resolve-build-config gained a new version step result (VERSION alone, no -<sha> suffix) to source the version label without re-parsing image-ref. Not yet extended to Go binaries/other non-container artifact types (see gap 24) or backfilled onto already-published images. |
| 24 | Go apps have no version-injection path. Unlike Node/Python/Maven, Go has no manifest-embedded version convention - build-image’s VERSION-resolution precedence chain falls through package.json/pom.xml/pyproject.toml for a Go app and lands on a VERSION file or literal 0.0.0, and even when a git tag resolves a real VERSION, nothing injects it into the compiled binary itself (no -ldflags -X anywhere in the build path) - payment-api --version (or equivalent) can’t report anything meaningful. |
Confirmed via build-image.yaml’s precedence chain: no go.mod-aware branch exists, and build-source.yaml/app Containerfiles have no version-injection convention. |
Fixed 2026-09-29: build-image.yaml now passes HANGAR_VERSION/HANGAR_GIT_REVISION as kaniko --build-args unconditionally on every build (not gated by a schema field - a Containerfile that doesn’t declare a matching ARG sees no effect). Deliberately generic rather than Go-specific: any language’s Containerfile can consume the same two args. See docs/user/cicd-yaml-reference.md’s new “Version metadata inside the image” section for the Go -ldflags -X recipe. Not yet exercised against a real Go app’s Containerfile - schema/render-verified only (helm template), no live TaskRun yet. |
| 25 | resolve-image-ref’s git-rooted fallback used to guess 0.0.0-<sha> when no git-tag was present, a ref that frequently didn’t match what build-image.yaml had actually published for that commit (real VERSION vs. a hardcoded placeholder), so a deploy/release could reference a genuinely unpublished tag and fail downstream with no clue this Task’s guess was the root cause. |
Code read of resolve-image-ref.yaml’s git-rooted branch, 2026-09-29 - no live incident yet, caught by inspection. |
Fixed 2026-09-29: the git-rooted-without-a-tag case now fails the Task immediately with an explicit error instead of guessing, enforcing the file’s own existing assumption (“every real git-rooted use case is tag-triggered anyway”). |
| 26 | No way to publish a non-container build artifact (a Python client wheel, a standalone Go binary) - only the primary container image had a publish path, so a team wanting to hand out a library/CLI alongside their service image had nowhere to put it. | Design gap noted while closing gap 24 (Go versioning). | Built 2026-09-29: new build.artifacts: [python-wheel|go-binary] schema field + build-artifact.yaml Task, publishing as a real OCI artifact (oras push) at <image-repo>:<version>-<type> - same registry/credentials the container image uses, version always sourced from build-image’s own resolved version result so the two never drift apart. oras added to the toolbox image (2026-09-29-oras-artifacts, pin bumped in all 3 chart defaults - also fixed a pre-existing drift where glidepath-control-plane/glidepath-app were still pinned to 2026-09-05-chain-slug while glidepath-catalog had already moved to 2026-09-27-release-file-schema, despite a comment on both saying they’re meant to stay in sync). Toolbox image build/push/smoke-test (oras + existing tools + baked-in schema all verified inside the built image before pushing) done locally via the container CLI; not yet exercised against a real Go/Python app’s PipelineRun - schema/render-verified only. Artifactory (or another real package registry, for pip install/go get semantics rather than just OCI-artifact storage) considered as an alternative publish target and deliberately deferred - needs an actual reachable instance decision first, see the Hangar Glidepath feature-work thread. |
| 27 | release-log-emit’s Loki logs carry service_name="unknown_service" live, not the glidepath value its own OTLP payload sets - found while verifying a platform-cicd-branding rename to release-log.json’s dashboard queries ({service_name="platform-cicd"}), which turned out to already be dead code matching nothing, before or after the rename. |
Live query against loki.observability.svc.cluster.local:3100/loki/api/v1/label/service_name/values on dev, 2026-09-29: only value returned is unknown_service. Ruled out catalog/lib/otel.sh’s otel_log_send as the cause - its resourceLogs[].resource.attributes already sets service.name: "glidepath" correctly (confirmed by reading it directly out of the live-pinned toolbox image, not just source). Root cause is one layer further down: Loki’s OTLP receiver only promotes an allow-listed set of resource attributes to real indexed labels (per release-log-emit.yaml’s own comment, service_name should be one of them) - whatever’s misconfigured is in Loki’s own OTLP-ingestion config (likely gitops-cluster-dev’s Loki Helm values), outside this repo. |
Not started - deliberately left alone this session rather than guess-fixing infra outside glidepath. release-log.json‘s two service_name="platform-cicd" query strings were left un-renamed (still platform-cicd, now provably dead either way) pending the real fix, so nobody mistakes a bareword glidepath there for a verified-correct query the way the mission-control/pipelines-overview dashboards’ now are. |
| 28 | CDEvents source prefix renamed /platform-cicd/ -> /glidepath/ (branding cleanup) - catalog/lib/cdevents.sh (baked into the toolbox image) and flow-triggers.yaml’s CEL filter (per-Application, Helm/ArgoCD-synced) are two independent rollout mechanisms, so a single-prefix cutover risked a real window where a build under an already-cached old toolbox image emits the old prefix while an already-resynced Trigger expects only the new one (or vice versa) - a successful build silently failing to chain to its next stage. |
Design review, 2026-09-29, before making the change live - not a live incident. | Mitigated by design, not yet fully closed: flow-triggers.yaml’s CEL filter matches BOTH /platform-cicd/... and /glidepath/... (deployed together with the toolbox rebuild emitting the new prefix, so there’s no window where only one side is aware of the other). Follow-up owed: once every node’s toolbox image cache is confirmed rolled over to glidepath-toolbox:2026-09-29-glidepath-source-prefix (no fixed date - depends on real cluster pull behavior), drop the /platform-cicd/ branch from the CEL filter as its own small cleanup PR. |
| 29 | platform-cicd-tenant-onboarding ApplicationSet still carries the old branding. Not renamed - confirmed live via kubectl that it owns 7 real Applications (backstage-cicd, baggage-api-cicd, boarding-api-cicd, checkin-api-cicd, flight-api-cicd, gate-api-cicd, skyport-broker-cicd) via ownerReferences. An ApplicationSet’s name is immutable, so a rename means delete+recreate; an unqualified delete cascade-deletes all 7 via Kubernetes garbage collection. |
Live kubectl get applicationset/kubectl get applications -o json query against dev, 2026-09-29. |
Not started - deliberately deferred, purely cosmetic value (name only ever seen via kubectl) against a real residual risk. If revisited: (1) add syncPolicy.preserveResourcesOnDeletion: true to the ApplicationSet as its own isolated, live-verified push first; (2) only then delete+recreate under glidepath-tenant-onboarding with an identical generator spec; (3) confirm ArgoCD re-adopts the 7 orphaned Applications rather than erroring/duplicating - this step wasn’t verified against this ArgoCD version’s actual behavior, only against documented intent. Separately, the backing GitHub repo name pattern (platform-cicd-<cluster>-tenants.git) is a real repo, not a k8s object - a repo rename is a bigger, separate decision, out of scope here. |
| 30 | platform-cicd-kind-dev ClusterSecretStore / Infisical project slug still carries the old branding. Not renamed - confirmed live (Valid, ReadWrite: True), backing a real external Infisical project (self-hosted or cloud) by that slug. No cascade-delete risk (unlike gap 29 - a ClusterSecretStore owns nothing), but the real blocker is external: renaming correctly requires renaming the actual Infisical project via Infisical’s own admin API/UI first (unclear if Infisical even supports renaming a project slug), which this session has no credentials for and shouldn’t handle directly per standing credential-handling preference. |
Live kubectl get clustersecretstore/kubectl get externalsecret -o json query against dev, 2026-09-29 - zero live ExternalSecrets found currently referencing this store by name, but that’s a point-in-time check, not proof nothing ever will. |
Not started - gated on an Infisical-admin-capable session renaming the actual project first. Once that’s done, the k8s-side follow-up (recreate ClusterSecretStore under the new name, re-provision <newSlug>-infisical-creds) is mechanical. |
| 31 | Top-level platform/ source tree (broker/dora-exporter/tekton-results-relay/sigstore) still carried old branding - all 3 Go modules declared module github.com/platform-cicd/..., and .github/workflows/broker-image-ci.yaml auto-published platform-cicd-token-review-interceptor on every push, meaning the next real interceptor commit would have silently undone the image rename from gap-29’s sibling work (item 3) without this being caught. |
Found while answering a direct question about this path, not from a prior audit - grep across platform/*/go.mod turned up the module declarations. |
Fixed 2026-10-03: directory moved to glidepath/, all 3 go.mod module paths + the 1 internal import renamed (go build ./... verified in all 3 modules), 4 Dockerfile comments + ~30 doc/comment pointers updated, and critically the CI workflow’s publish target and path filters fixed. docs/archive/ left untouched as always. Added .gitignore to each Go module - this session’s own verification builds wrote stray compiled binaries into the module root with nothing catching it before. |
| 32 | PR-comment idempotency markers still carried old branding (<!-- platform-cicd:gate:<name> --> in comment-pr-check-result.yaml, <!-- platform-cicd:preview-env --> in glidepath-pr-preview-notify’s notify-job) - a hard single-marker cutover risked a duplicate comment on any PR with an already-open old-marker comment (the new Task version wouldn’t recognize it as the same comment to edit in place). |
Design review, 2026-10-03, before making the change live - not a live incident (worst case is cosmetic: one duplicate comment per affected in-flight PR, not a broken pipeline). | Fixed with a dual-accept lookup, same pattern as gap 28’s CDEvents source-prefix migration: both files now write only the new glidepath:... marker, but the existing-comment lookup matches either marker, verified against synthetic comment data for all 3 cases (old-marker found, new-marker found, neither found). Follow-up owed: drop the old-marker branch from both lookups once confident no PR with an old-marker comment is still open - no fixed date, same reasoning as gap 28. |
| 33 | Glidepath only ever deploys to this cluster’s own Argo Rollout - no way to target AWS ECS, Lambda, or any non-k8s platform, a real limitation for any app that doesn’t belong on this cluster. | Design review - the original Phase 6 proposal that started this whole branding/versioning/artifact-type work thread. | Built 2026-10-03 (ECS first): deploy.target: aws-ecs + deploy.ecs.{cluster,service,containerName,region,taskDefinitionFamily} schema fields, a new deploy-ecs.yaml Task (registers a new task-definition revision with the built image, updates the service, waits aws ecs wait services-stable), and resolve-deploy-target.yaml to drive a when: gate in deploy.yaml between it and the existing deploy-manifests path. Uses the official public.ecr.aws/aws-cli/aws-cli image (confirmed real multi-arch + bundled jq via container image pull/inspect/run, not assumed) rather than adding AWS CLI to the Alpine/musl toolboxImage. AWS credentials reuse this app’s own existing secrets: mechanism (aws-access-key-id/aws-secret-access-key), not a new credential path. The image-patch jq transform and the container-name-not-found guard were both tested against synthetic ECS task-definition JSON before trusting them. Not live-verified - no AWS account/credentials exist in this environment; schema/render-verified only (helm template). A git-rooted deploy (no build/test in that run) resolves deploy.target correctly too, not just the event-chained path - fixed same day by having resolve-deploy-target.yaml clone just cicd.yaml directly via the GitHub App token broker when config-json is unpopulated, the same authenticated pattern deploy-manifests.yaml already uses (deliberately NOT resolve-notify-config.yaml’s own git-rooted fallback, which clones unauthenticated - acceptable for a best-effort Slack message, not for a result that decides which deploy Task actually runs). Phase 6 completed 2026-10-04: added aws-lambda (deploy-lambda.yaml - updates an existing function’s container image via aws lambda update-function-code + waits on the function-updated-v2 waiter, confirmed that waiter actually exists in the pinned CLI via aws lambda wait help, not assumed; hard-fails with a clear error if image-ref isn’t an ECR URI, since Lambda cannot pull from an external registry like ghcr.io at all - a real AWS constraint, not a Glidepath gap) and azure-container-apps (deploy-azure-container-apps.yaml - az containerapp update --image + a manual poll loop on properties.runningState, since unlike ECS/Lambda there’s no single blocking wait subcommand for this; uses the official mcr.microsoft.com/azure-cli image, confirmed real multi-arch + bundled jq the same way the AWS CLI image was). Also fixed a second instance of the exact same git-rooted bug while wiring these in: deploy.yaml’s three cloud-target task entries were still sourcing config-json from the raw $(params.config-json) (literally "{}" for a git-rooted deploy) instead of resolve-deploy-target‘s own already-resolved copy - so even though resolve-deploy-target would have correctly identified aws-ecs for a git-rooted deploy, deploy-ecs.yaml itself would still have found no cluster/service/containerName to act on. Fixed by having resolve-deploy-target output the resolved config-json as a second result, reused by all three cloud-target Tasks instead of each re-deriving (and re-breaking) it. All three cloud targets’ credential and config extraction logic tested against synthetic data before trusting it (ECR-URI regex guard, Lambda/Azure config-json extraction). None of the three are live-verified - no AWS or Azure account exists in this environment; schema/render-verified only (helm template). |
| 34 | Onboarding scaffolded platform/pr-env.yaml only when ephemeralEnvironments.pullRequest.enabled was already true, and never platform/base.yaml or platform/envs/<lower-env>.yaml. The Airframe starter cicd.yaml enables no previews, so a freshly onboarded app (found end-to-end on sky-marshall, 2026-10-03) had none of the platform/ files. The developer hand-copied them from another app, including a dev.release.yaml carrying that app’s image, which (release file wins the merge) made sky-marshall’s dev run boarding-api’s image, and made Tower resolve the wrong source repo from its provenance. |
jfillman/sky-marshall at 57e5f43 (hand-added platform/ files; dev.release.yaml pinned ghcr.io/jfillman/boarding-api), live app-sky-marshall-dev Rollout running that image on dev. |
deliver-onboarding-files now scaffolds all three, once each and only when missing, for every app (platform/base.yaml as {} with a comment, platform/pr-env.yaml comment-only, one platform/envs/<env>.yaml with envName per deploy.lowerEnvironments, default dev). Rendered the Task and ran the scaffold block against the real sky-marshall cicd.yaml in a scratch repo (creates 3 files, 2nd run no-ops). Not yet exercised by a live onboarding-resync run. |
| 35 | A change to cicd.schema.json reached main but not the running toolbox, so every cicd.yaml using deploy.target failed validation. The schema is baked into the toolbox image, and Phase 6 added deploy.target / deploy.ecs / lambda / azureContainerApps without a rebuild. deploy has additionalProperties: false. |
validate-config on smoke-ecs: deploy: Additional property target is not allowed. Pulling the live image 2026-09-29-glidepath-source-prefix and grepping its schema: 0 mentions of aws-ecs, target or taskDefinitionFamily. |
Rebuilt multi-arch as 2026-10-04-deploy-target-schema (only schemas/cicd.schema.json differs from the previous image, diffed over every file under /opt/platform) and bumped toolboxImage in all three charts together (glidepath#7). Nothing yet stops this from recurring: a CI check that fails when schemas/cicd.schema.json changes without a toolboxImage bump would. |
| 36 | deploy-ecs could not deploy any task definition missing an optional field, and dropped fields it did not list. It copied a fixed list of fields out of describe-task-definition; an unset one (taskRoleArn on a definition with only an execution role) became JSON null, which the CLI rejects before calling AWS. Fields outside the list (runtimePlatform, ephemeralStorage, tags, pidMode, …) were silently dropped, so an ARM64 Fargate task would have come back as X86_64. |
First live run (smoke-ecs, 2026-10-04): Invalid type for parameter taskRoleArn, value: None. Reproduced offline against the real definition by running the old projection through the CLI’s client-side validation. |
Now re-registers the whole definition with only read-only fields removed and nulls stripped. Verified: the shipped jq filter run against the real definition and a synthetic one with every optional field; the CLI’s validation accepts the result; nothing was registered during testing. Not yet verified: the live register-task-definition / update-service / wait services-stable sequence. |
| 37 | Go builds for amd64 crash on the arm64 build node. The amd64 half of build-image runs under qemu-x86_64 on dev (arm64, kernel 6.12.28) and the Go 1.23 compiler dies: runtime: lfstack.push invalid packing ... fatal error: lfstack.push, exit 2. binfmt is healthy, so emulation works; this Go version does not survive it. The GoApplication starter Containerfile (RUN CGO_ENABLED=0 go build) triggers it for every new Go app that builds both platforms. |
smoke-ecs build step-build-and-push-amd64 exit 2 while step-build-and-push-arm64 completed. |
Worked around per app with build.platforms: [arm64] (valid only for an app that never deploys to prod). Fixed for Containerfiles that opt in (2026-10-17). build-image now uses BuildKit, which honors FROM --platform=$BUILDPLATFORM: compile in a build-platform stage with GOOS=$TARGETOS GOARCH=$TARGETARCH and the compiler never runs emulated (kaniko ignored the per-stage platform, so this didn’t work before). The GoApplication scaffold does this from airframe’s next release. A Containerfile that compiles in a target-platform stage still emulates and still crashes - existing Go apps scaffolded before then need the same two-line change (see cicd-yaml-reference.md). |
| 38 | aws-lambda could not work at all: the build always pushes to ghcr.io and Lambda only pulls from ECR. deploy-lambda guarded for an ECR image-ref and failed otherwise, but nothing in the pipeline let an app set one (deliver-onboarding-files.yaml hardcodes image_repo=ghcr.io/<owner>/<app>), and Lambda also rejects the multi-architecture image index build-image publishes. |
Found by reading the code while preparing the first Lambda run, then confirmed by testing the real copy: crane copy of smoke-ecs:0.0.0-4eedae7-arm64 into an ECR repository succeeded with identical source and destination digests. |
deploy-lambda now copies the per-architecture image into the function’s current ECR repository and deploys the ECR digest (see the Task header for why this and not an ECR build). Registry-touching Tasks other than this one still only speak ghcr.io. Not yet verified: the Task running live against a real function. |
| 39 | Future feature: Ground environments on other clusters. A Ground environment may not set cluster; every Ground environment runs on the control plane’s cluster. Wanted (several dev clusters run by one Glidepath), deliberately not built yet. |
Validator refuses it (glidepath-app validateEnvironments); requirements Q3. |
ADR-0024 “Future”: allow a lower, workloads cluster. First step, useful alone: deploy-manifests waits on Rollout health through the relay’s facts instead of same-cluster RBAC. Then per-cluster Argo CD for the Ground ApplicationSet, secrets store and pull credentials, tests, Tower. |
| 40 | Two cluster registries. Glidepath’s (glidepath-control-plane clusters:) and Airframe’s (crossplane-system ConfigMaps, type: dev|upper) described the same clusters in different words, plus apron’s type; they had drifted (an unused dev duplicate of kind-dev). |
ADR-0024 Consequences. | Fixed 2026-10-08: one record per cluster in the hub cluster repo’s clusters.yaml, read by Glidepath’s control plane (value file) and Airframe’s charts/cluster-registry (ADR-0024 amendment). Remaining (deferred by the owner, 2026-10-08): the Composition gates still read the derived type. When multi-cluster Ground (#39) or a second lower workload cluster lands, move them to roles, not zone: the dev-cluster gate means control-plane, the ApplicationEnvironment gate means workloads (a zone check would wrongly refuse a Flight environment on a lower workload cluster). Then drop type. |
| 41 | Image gates failed on every Release Record PR, and full-cloned the gitops repo to read one file. release-record-* branches matched the gate templates’ startsWith("release-"), so sast/image-scan/sbom/provenance ran on PRs that promote nothing and failed (“no */release.yaml changed”); on real release PRs each gate cloned the whole repo for a HEAD~1 diff, which also broke on any branch with a second commit. |
135 of 284 extract-promoted-image runs in Tekton Results failed, all on release-record-* PRs (2026-10-09); REPORT-glidepath-pipeline-performance-2026-10-08.md item 5. |
Fixed 2026-10-09: templates exclude release-record- branches; the image gates resolve the manifest through the GitHub API (resolve-promoted-image StepAction, PR base…head compare); verify-commit-signature and validate-values read the PR over the API too. Gitops repos need an onboarding resync to pick up the template change. |
| 42 | No image promotion exists. The image-promotion release gate was a stub whose only real part polled the Checks API until the other gates reported, then no-op’d: its result was the AND of its siblings, it could not block a merge (GitHub Free enforces no rulesets on private repos), and every cluster pulls from ghcr.io so there is no upper registry to copy into. 97 runs on real release PRs: 21 passed, 73 failed because a sibling failed (a duplicate failure comment each), 3 ran the full 18 min. |
Tekton Results archive, 2026-10-09; gh api .../rulesets -> 403 “Upgrade to GitHub Pro”. |
Retired 2026-10-09 (ADR-0025): template, Pipeline and registry entry removed; wait-for-release-guardrails kept generic. A real promotion is ordered by registry topology: pre-merge with event-driven fan-in on the release record if a separate ArgoCD-pulled upper registry exists, post-merge in bypass-merge-check if the registry is shared. Superseded by ADR-0029 (2026-10-27 to 10-30): Ground and Flight registries; a real, required image-promotion check started by an event fan-in when the last required gate is green; hangar-prod admits only promoted Flight images (enforced 2026-10-30). See registries.md. |
| 43 | Azure deploys still use a stored secret. Lambda and ECS deploys are keyless since 2026-10-16 (one IAM role per function or service, assumed with a one-hour token); Azure Container Apps still logs in with a stored service principal secret. | cloud-deploy-identity.md, “Not covered yet”. |
AWS closed 2026-10-16: deploy-lambda (phase 4) and deploy-ecs have no stored-key path and mount no app-secrets; ECS also copies the image into the target’s ECR repository, so a private package deploys. Next: Azure through workload identity federation. |
| 44 | Glidepath events are not durable. GET /v1/events is SSE from each glidepath-api replica’s in-memory buffer (2000 events, 24 h). When every replica restarts at once the buffer is empty, and a subscriber that is down longer than the buffer reaches back misses events with no way to know. Autopilot’s trigger bridge, the one unattended consumer, can miss a trigger. There are no acks and no per-consumer queues. |
api.md#events “Best-effort, not durable”; ADR-0030 (proposed 2026-11-01, glidepath#214). | Deferred (user, 2026-11-01): come back to it. ADR-0030’s design: a RabbitMQ topic exchange glidepath.events beside SSE. glidepath-api publishes once per event with confirms; each unattended consumer gets its own quorum queue, bindings and user declared in Git; Autopilot moves first with SSE as its fallback. Pick up when a second unattended consumer appears, or when a missed trigger costs something. |