CI/CD · Glidepath as the worked example
The guarded descent from a merged commit to a verified release
The best CI/CD is the kind developers hardly think about. Glidepath, Hangar's CI/CD platform, runs on Tekton and Pipelines-as-Code on plain Kubernetes. Developers write one file, and the platform takes care of the rest: triggering, chaining, tracing, governance and promotion. Each practice below comes from a decision record I wrote at the time, so you can see the reasoning, including the parts that didn't work the first time.
A developer's pipeline should be one file, and nothing in it should ever reach a production cluster except a reviewed commit.
The contract and the engine
Give developers one file, and own everything behind it
Developers configure their pipeline through
cicd.yaml, read fresh from the triggering commit and validated against a JSON Schema with readable errors. They never write Tekton YAML. The engine is a fixed superset DAG with stages switched on by config, not a compiler for arbitrary graphs, which keeps every pipeline supportable.Scaffold the config instead of documenting it
A new app gets a real, minimal
cicd.yamlcommitted at onboarding, so its first push runs a pipeline. Building this surfaced a schema that had silently fallen behind the app stacks, which is exactly the kind of drift a scaffold catches and a docs page does not.Chain stages with events, not with one giant pipeline
Build, test and deploy are separate PipelineRuns. A shared broker relays CDEvents between them, and each tenant's next stage runs under that tenant's own least-privilege identity, never the broker's.
Version the shared catalog, and upgrade tenants by moving a pin
The shared Tekton catalog is a Helm release pinned by git tag. Moving a tenant to a new catalog is a reviewed change, and a canary tenant can run a catalog branch first. The better end state (an OCI bundle referenced by digest) is written down as not built yet, rather than rejected.
Build and test
Build images without privilege
Kaniko under Pod Security Standards
restricted: no privileged daemon, no host namespaces, on any conformant cluster. It was validated against the target cluster before any app depended on it, with rootless buildah as a real fallback.Read the source when the docs disagree
Testkube's documented multi-namespace mode crashed the API server: the source code gates it to a paid edition. The shipped design runs every tenant's tests in one shared namespace and moves each app's secrets in only for the duration of a run.
Close with policy what RBAC cannot express
RBAC controls who can edit a Secret, not which Secret a pod may mount. An admission policy (Kyverno ValidatingPolicy, CEL) closes that gap at the one point where the real tenant identity is still visible.
Preview environments ride the production chart
Per-PR environments deploy through the same chart every other tier uses, tag images by commit sha only (so a force-push can never serve a stale image under a reused tag), and are cleaned up by a TTL sweep, a trade-off chosen on purpose.
Security and supply chain
Make unfinished gates loud
SAST, image scanning, policy and SBOM exist as explicit extension points from the start. A stub is marked as a stub in its result and on the dashboard, and is never reported as a real check, because a gate that silently passes is worse than none.
Sign keylessly, with the trust root that fits the identity
Commits are signed against public Sigstore, which already trusts a human's GitHub or Google identity. Images and provenance are signed against a self-hosted Fulcio that trusts the cluster's own service-account issuer, with a self-hosted transparency log beside it.
Verify what the build did, not only who signed it
Release policy evaluates the SLSA provenance attestation itself (which tasks really ran for this image), in addition to the commit signature. The two checks answer different questions, and both are required.
Keep pipeline permissions and running workloads apart
Every app gets a CI namespace and one namespace per environment, as peers. Pipelines execute in one; the service runs in the others; they never share a blast radius or fight over names.
Store no credentials you could avoid storing
Every chart gets secrets through External Secrets from Infisical, authenticated with the cluster's own TokenReview. App-owned secrets are referenced where they live, not mirrored: the mirror turned out to be wider than the original, a real least-privilege regression.
Release and multi-cluster
Release is a pull request, and ArgoCD is the only writer
The release stage never touches a cluster. It opens a PR against the app's GitOps repo with the exact digest that passed test and deploy, never rebuilt. Gates are required checks, a human reviews, and ArgoCD's sync is the only thing that changes the target.
One ArgoCD per cluster, outcomes back as events
A single ArgoCD holding credentials for every cluster is a path from dev into prod. Each cluster runs its own, and reports back through sync hooks that fire only when a release really converges. ArgoCD Notifications was tried first and dropped after live testing showed it fired on drift correction too.
Keep cluster state out of the platform's repo
Per-cluster trust roots and identity are generated per cluster and written into that cluster's own repo, never shared and never committed to the platform's. The platform stays installable on clusters that do not exist yet.
Operations
Keep the history, not just the metrics
Metrics say how long runs took; they do not say what a run did. Tekton Results archives the full runs and logs, and becomes the main cleanup path, with the pruner left as a safety net.
What is still rough
A CI/CD platform that claims to have no gaps is not being used. Glidepath keeps aknown-gaps list of problems found building real apps on it, each with its evidence and a direction: webhook delivery that once failed for hours without an alert, required checks that assume every PR is a release PR, and unsigned agent commits failing the provenance gate. Those are the next things to fix, and they are listed in public on purpose.