CI/CD · Glidepath as the worked example
The guarded descent from a merged commit to a verified release
The best CI/CD is the kind developers hardly think about. Glidepath, Hangar's CI/CD platform, runs on Tekton and Pipelines-as-Code on plain Kubernetes. Developers write one file, and the platform takes care of the rest: triggering, chaining, tracing, governance and promotion. Each practice below comes from a decision record I wrote at the time, so you can see the reasoning, including the parts that didn't work the first time.
A developer's pipeline should be one file, and nothing in it should ever reach a production cluster except a reviewed commit.
The contract and the engine
-
Give developers one file, and own everything behind it
Developers configure their pipeline through
cicd.yaml, read fresh from the triggering commit and validated against a JSON Schema with readable errors. They never write Tekton YAML. The engine is a fixed superset DAG with stages switched on by config, not a compiler for arbitrary graphs, which keeps every pipeline supportable. -
Scaffold the config instead of documenting it
A new app gets a real, minimal
cicd.yamlcommitted at onboarding, so its first push runs a pipeline. Building this surfaced a schema that had silently fallen behind the app stacks, which is exactly the kind of drift a scaffold catches and a docs page does not. -
Own the folder, not the chart
Glidepath owns one folder in the app repo,
glidepath/, with a base file and one values file per environment, and treats what is inside as opaque Helm values except the release keys it writes itself. The chart that reads them is per-app configuration behind a written chart contract, with Airframe's application chart as the reference implementation, so a team can bring its own. -
Define every environment once
An app's environments are one list in
cicd.yaml: a name, a tier (Ground or Flight), and a cluster or cloud target. Order is promotion order, steps refer to environments by name, and every cluster artifact is derived from that list rather than kept beside it. Tower edits the list as a pull request, with a preview of every file it touches. -
Let an app pick its chart, per environment
An app can name its own chart, from git or a registry, for every environment or just one, over the cluster's default, so a new chart version can be tried in dev before staging sees it.
-
Chain stages with events, not with one giant pipeline
Build, test and deploy are separate PipelineRuns. A shared broker relays CDEvents between them, and each tenant's next stage runs under that tenant's own least-privilege identity, never the broker's.
-
Version the shared catalog, and upgrade tenants by moving a pin
The shared Tekton catalog is a Helm release pinned by git tag. Moving a tenant to a new catalog is a reviewed change, and a canary tenant can run a catalog branch first. The better end state (an OCI bundle referenced by digest) is written down as not built yet, rather than rejected.
Build and test
-
Build images without privilege
Rootless BuildKit, one daemon per build: UID 1000, no privileged container, no host namespaces, Pod Security
baseline. Both architectures build in one step, and a$BUILDPLATFORMstage cross-compiles instead of emulating. A spike measured every claim on the live cluster before the switch, including thatrestrictedis out of reach for any rootless builder that needs a user namespace. -
Read the source when the docs disagree
Testkube's documented multi-namespace mode crashed the API server: the source code gates it to a paid edition. The shipped design runs every tenant's tests in one shared namespace and moves each app's secrets in only for the duration of a run.
-
Close with policy what RBAC cannot express
RBAC controls who can edit a Secret, not which Secret a pod may mount. An admission policy (Kyverno ValidatingPolicy, CEL) closes that gap at the one point where the real tenant identity is still visible.
-
Preview environments ride the production chart
Per-PR environments deploy through the same chart every other tier uses, tag images by commit sha only (so a force-push can never serve a stale image under a reused tag), and are cleaned up by a TTL sweep, a trade-off chosen on purpose.
Security and supply chain
-
Make unfinished gates loud
SAST, image scanning, policy and SBOM exist as explicit extension points from the start. A stub is marked as a stub in its result and on the dashboard, and is never reported as a real check, because a gate that silently passes is worse than none. By October 2026 seven gates are real, including a check ride and API compatibility, and admission-policy validation is the one stub left.
-
Retire a gate that can never be honest
The image-promotion check waited for every other gate and then promoted nothing, because one registry serves every cluster. Rather than keep a green check that meant nothing, it was removed, and the decision records where a real promotion would live once the registry topology needs one.
-
Sign keylessly, with the trust root that fits the identity
Commits are signed against public Sigstore, which already trusts a human's GitHub or Google identity. Images and provenance are signed against a self-hosted Fulcio that trusts the cluster's own service-account issuer, with a self-hosted transparency log beside it.
-
Verify what the build did, not only who signed it
Release policy evaluates the SLSA provenance attestation itself (which tasks really ran for this image), in addition to the commit signature. The two checks answer different questions, and both are required.
-
Prove the merge, not just the signature
A commit made through GitHub's merge button carries GitHub's key, not a person's. Glidepath accepts it only on a separate path: a merged PR into the default branch, merged by an allowed login, with optional approval counts checked by the gate itself. Who may sign is a policy platform admins own per cluster, not a field in the app's own file.
-
Keep pipeline permissions and running workloads apart
Every app gets a CI namespace and one namespace per environment, as peers. Pipelines execute in one; the service runs in the others; they never share a blast radius or fight over names.
-
Store no credentials you could avoid storing
Every chart gets secrets through External Secrets from Infisical, authenticated with the cluster's own TokenReview. App-owned secrets are referenced where they live, not mirrored: the mirror turned out to be wider than the original, a real least-privilege regression.
Release and multi-cluster
-
Release is a pull request, and ArgoCD is the only writer
The release stage never touches a cluster. It opens a PR against the app's GitOps repo with the exact digest that passed test and deploy, never rebuilt. Gates are required checks, a human reviews, and ArgoCD's sync is the only thing that changes the target.
-
One ArgoCD per cluster, outcomes back as events
A single ArgoCD holding credentials for every cluster is a path from dev into prod. Each cluster runs its own, and reports back as events. The first version used sync hooks that fired only when a release really converged; since October 2026 the Rollout itself sends the facts (ADR-0021).
-
Keep cluster state out of the platform's repo
Per-cluster trust roots and identity are generated per cluster and written into that cluster's own repo, never shared and never committed to the platform's. The platform stays installable on clusters that do not exist yet.
-
Give cloud production the same reviewed approval
A Lambda function or an ECS service has no GitOps repo, so its approval is a pull request on its own repo that moves a per-environment release pin to an exact digest. The same gates run on it, and the deploy happens only after it merges. Before this existed, Glidepath refused to give a cloud service a Flight environment at all.
-
Prod sends facts, the control plane decides what they mean
Each cluster's Argo Rollouts posts a small fact on every phase change. The control plane joins it to the release's tracking record and decides whether a release is healthy, failed or rolled back. Outcome hook Jobs, their extra identity and their cross-cluster address all went away.
-
Zone, roles and tier are three different things
A cluster has a zone (lower or upper: who may write to it) and roles (what the platform installs there). An environment has a tier (Ground or Flight: how a change reaches it) and may be marked production. Rules that used to hide in one overloaded word are now separate checks: a production environment is Flight, and on Kubernetes it runs on an upper cluster.
Operations
-
Keep the history, not just the metrics
Metrics say how long runs took; they do not say what a run did. Tekton Results archives the full runs and logs, and becomes the main cleanup path, with the pruner left as a safety net.
What is still rough
A CI/CD platform that claims to have no gaps is not being used. Glidepath keeps a known-gaps list of problems found building real apps on it, each with its evidence and a direction: webhook delivery that once failed for hours without an alert, required checks that assume every PR is a release PR, and unsigned agent commits failing the provenance gate. Those are the next things to fix, and they are listed in public on purpose.