A CI/CD engine you can call

On October 30, 2026, I took away Tower's permission to create a Tekton PipelineRun. Then I opened a gate-api build in Tower and clicked Re-run.

The run started. Tower couldn't have started it any more, and it didn't try. Instead, it asked Glidepath to re-run the build for me, and Glidepath wrote one line to its audit log: operation rerunRun, status 201, caller tower-actor, on behalf of jfillman. Ten minutes earlier I'd cancelled a checkin-api build the same way. Same kind of line, same two names.

That small moment is what this essay is about. Glidepath, Hangar's CI/CD engine, used to be a set of Tekton pipelines that only Tower knew how to drive. It's now an API that anything can call, Tower included, and that decides for itself who may do what. I want to explain how it works, and argue that it's the right shape for a CI/CD engine.

Tower was the API, and that was the problem

Until late October, everything a person could learn about Glidepath went through Tower's backend. One file in Backstage held about sixty routes. They read run history from Tekton Results, test runs from Testkube and signatures from Rekor. They verified image signatures, and they wrote PipelineRuns straight into the cluster through a Kubernetes proxy whenever somebody clicked Re-run or Cancel. It worked well, and I'd built most of it on purpose.

But it meant Glidepath's real API was a Backstage plugin. An agent can't call a Backstage route. A script would have to pretend to be Tower. A second portal would have to copy all sixty routes and get them all right. And the most important rule in the whole system, who may re-run whose build, lived in a portal's backend, where the CI/CD engine couldn't see it or enforce it. Glidepath couldn't stand on its own.

A portal should be a client of the CI/CD engine, not the place where its rules live.

Why it had to be an API, again

A week earlier, in Paving the cloud, I argued that a paved road has to be an API. A form in front of a ticket queue can't be versioned, reviewed, undone or used by an agent; an API can. An IDP that's only a portal is a catalog of things you can request, and an IDP with an API underneath is a catalog of things the platform will keep true.

That essay was about nouns: a queue, a bucket, a database. For nouns, the Kubernetes API is a wonderful home. You get a schema, a status, and a controller that puts things back when they drift. CI/CD is mostly verbs. Run this again. Why did it fail? Show me the log. Promote it to staging. Roll it back. In What Hangar costs I drew a line between things that should be reconciled and things that should just happen once, and a re-run is clearly the second kind. Nobody wants a controller re-running their build forever.

It's tempting to say the API already existed, because a PipelineRun is a Kubernetes object. But a PipelineRun is the implementation. It has more fields than anyone should fill in by hand, and if every client may create one, every client may create any one, with any service account and any task. Giving Tower that right is how I ended up needing admission policies to keep Tower honest.

So glidepath-api is plain HTTP that borrows Kubernetes where Kubernetes is good, which is identity and access control, and keeps the verbs at the level people think in. Built that way, an API-driven CI/CD engine buys a handful of things a portal backend can't:

• One front door. Tower, an agent, a script and a person with curl get the same answers to the same questions.
• One place to decide, and one place to record. Every call is authorized and written down by the engine, whoever is asking.
• A contract. Every operation, with what it risks, is in one file an agent can read and a test can check.
• Narrow grants. Tower's identities can no longer write a PipelineRun at all. The engine holds that right, behind its own rules.
• A replaceable portal. Glidepath stands alone. Tower is one client, and it doesn't have to be the last.

What it is

glidepath-api is a small Go service: two replicas on the control-plane cluster, behind one Service on port 8080. Every merge that touches it builds a new multi-arch image, and the contract is baked into the image, so GET /v1/contract always describes the code that's answering. It has 30 operations, and the first thing to know about each one is its tier:

• T0, read. 26 of the 30. Nothing changes.
• T1, reversible in Ground. Re-run and cancel a build in the app's own CI namespace.
• T2, propose for Flight. Promote and roll back a production-side environment. The API only starts the app's release pipeline; the change arrives as a GitOps pull request that a person merges.
• T3, never offered. Anything that would change production directly.

Two things are missing on purpose. Promoting to a Ground environment and pinning a cloud release are both commits to a repo, so they need a GitHub write identity, and the API doesn't hold one. They stay in Tower for now. Here's the whole surface:

A few of those endpoints deserve an extra mention, because they're things Tower never had.

Explain. A failed run is classified against Glidepath's list of known failures, in the contract's order, with the line in the log that matched. When I added the last three entries, every one of the twenty most recent real failures on the dev cluster came back with a name and a hint.

Pushed, not polled. Tower's CI/CD tab used to ask for live runs every six seconds, which is about 760 calls an hour for each open tab. live/stream is one connection: a snapshot, then each change. Behind it, one watcher per namespace serves every viewer of that namespace and stops when the last one leaves. The event stream works the same way for CDEvents: every build, gate and release event the shared broker carries, re-verified, as server-sent events that resume from the last id you saw.

Validate before you propose. cicd/validations takes a change to an app's cicd.yaml and says whether it's valid, which fields it touches, and whether an agent would be allowed to make it. It's the same validator as the new cicd-validate pull-request check, which now runs on all six app repos, so the answer before the pull request is the answer the check will give. The first real agent pull request through it rewrote all 73 lines of the file, because the API returned re-marshalled YAML. Now it edits the file's own text, and a one-line change is a one-line diff.

Who may do what

I didn't want glidepath-api to invent its own users, roles or tokens. The cluster already knows who everyone is, and Kubernetes RBAC is already the thing my security reviews read. So the API asks the cluster twice.

Who are you? is a TokenReview. Anything the API server trusts works: an in-cluster ServiceAccount, or a federated identity from the other cluster, which is how Tower's identities from the prod cluster arrive. (Since November 1 there's one more way in, for people. That gets its own section below.)

May you? is a SubjectAccessReview in the app's CI namespace, on an API group that nothing actually serves: glidepath.hangar.io. Its resources are runs, runs/logs, releases, evidence, tests, environments, events and cicd. RBAC only needs the strings, so access is an ordinary ClusterRole. There are three: a reader, an operator who may re-run and cancel, and a releaser who may promote and roll back. They hold nothing of each other's, and all three are bound to nobody on a fresh install. To let one identity into one app, you bind the role in that app's namespace. And an app that doesn't exist answers 403 to anyone who couldn't read it if it did, so the API never tells a stranger which apps exist.

For whom? Tower acts for people, and people in Backstage aren't Kubernetes identities, so no SubjectAccessReview can be asked about them. A listed delegate, Tower or Clearance, may name the person in an X-Glidepath-On-Behalf-Of header. Every write from a delegate must name someone, or it's refused before anything is read. The API authorizes the delegate and records the person, and it trusts the delegate to have checked that person, which Tower does with its owner permission before it calls. I want to be clear that this is a trust I chose, not one the API can prove. The audit line is what keeps it checkable.

The cluster's own lock. glidepath-api's ServiceAccount has to be able to create PipelineRuns in any app's namespace, because apps are onboarded after it's installed, and RBAC can't narrow that to "only Glidepath's kind of run". So an admission policy does. It holds the API to Glidepath's shape: an app's or InfraService's CI namespace, a pipeline from the platform catalog, the standard runner, no inline spec, and an update may only cancel. The same policy refuses any PipelineRun write from Tower's writer identity. If the API ever had a bug that built the wrong run, the cluster would still say no.

Safe to retry. Every write takes an Idempotency-Key, and the run's name is derived from it, so sending the same request twice returns the first run with a 200 instead of starting a second one. Every write also has a dry run. And every refusal is JSON with a stable rule, like GP-AUTHZ-001 or GP-API-422, and a hint about what would work. A person reads the hint; an agent branches on the rule.

No GitHub credential. Promotions and rollbacks have to read the app's repos. glidepath-api holds no GitHub token at all. For each request it asks Glidepath's broker for one, and the broker mints a read-only installation token scoped to that one repository, and only if the repository really belongs to the app. The token lives for the request. It's never cached, logged or returned.

Moving Tower without a flag day

The riskiest part of an API like this isn't the API. It's moving the existing client onto it without breaking the views people use every day. So Tower moved one group of routes at a time, and each group went through three modes: direct, the old code; shadow, where Tower still answered from its own code but also asked the API in the background and logged every field that differed; and api, where the API's answer was the only one.

The run history caught the first real problem. A busy app's first read from Tekton Results could take a few seconds, Tower gave up at ten, and the API threw the half-finished read away. So nothing got cached, and every retry started cold. Now a read that fills the cache finishes even when its caller has gone, callers asking the same thing share one read, and each replica warms every app's last seven days when it starts. The slowest tenth of history calls fell from 10 seconds to under 7, and the errors went to zero.

The evidence comparison caught something better. To move it, I ported Tower's signature verification to Go by hand, so its answers would match field for field. Doing that, I noticed that the identity check was an unanchored regular expression. A signature from a branch called main-evil would have passed a policy that said main. Both implementations now anchor the match. I'd reviewed that code before and missed it. Writing it a second time didn't. That's a merit of an API-driven design I didn't expect: the second client is also a code review of the first.

On October 30 the old paths were deleted. Tower now asks glidepath-api for every run, log, release record, piece of evidence and test run, and sends every re-run, cancel, promotion and rollback through it. There is no fallback. If the API fails, Tower shows the error, with the trace id in it. That afternoon I rolled gate-api's production back to an earlier version from Tower. The audit log showed the rollback plan read on my behalf, then rollbackEnvironment 201 from tower-actor. A release run opened the pull request, I merged it, and the release record went from progressing to healthy.

Watching the API itself

If every CI/CD click in Hangar now goes through one service, that service had better be the best-watched thing in the platform. On October 31 it got the same treatment I'd give a product.

Metrics. Each replica serves Prometheus metrics on its own port, with nothing else on that listener: requests by operation, tier, status and decision; how long each took; open streams. A caller that gives up before the answer is recorded as a 499, not a 502, because that's what really happened.

SLOs. Two, over 30 days, through Sloth: 99.5% of calls don't fail, and 95% of calls answer within 2.5 seconds. On the dev cluster the latency SLO started burning on day one, at about 1.8 times the budget, because the API server answers TaskRun lists slowly there. I left the objective where it was. An SLO moved to make an alert go away isn't measuring anything any more. Four more alerts cover what an SLO can't: no replica scraped, a replica missing, and each kind of stream close to its limit.

Traces. Each call is a span named for its operation, with the app, the tier, the caller and the decision on it, and each thing the API reads for that call is a child span. A caller that sends a W3C traceparent gets its trace joined. Prod Backstage was already instrumented with OpenTelemetry, and I checked that a traceparent survives the API server's proxy unchanged, so a click in Tower on one cluster and the Tekton Results read on the other end up in one trace. The trace id comes back in a response header, it's in the audit line, and Tower ends its error messages with it.

Dashboards. Grafana has a new Glidepath API dashboard: both SLOs, their budget and burn, the alerts, traffic and latency per operation, open streams, and Tower's own measurement of its calls, read from the prod cluster, next to the API's. The gap between the two lines is the network between the clusters, which is the number I'd been guessing at for a month. The pipelines dashboard got PipelineRun durations and outcomes, runs waiting to resolve, pod start latency and the traces of failed and slow stages. It also turned out to be querying Tekton metric names that hadn't existed for several releases. Those are fixed.

A door for people

Until November 1 a person could only reach the API through the Kubernetes API server's proxy, which means holding a cluster credential. That's fine for me and wrong for everyone else. So glidepath-api now has a route on the cluster's Gateway, serving /v1 and nothing else, and it accepts a person's ID token from people-dex, the same Dex in front of GitHub that Clearance already trusts.

The API checks that token itself. A token goes down that path only if its issuer is people-dex, and then it's fully verified: the signature against Dex's keys, the issuer, the expiry, and the audience, which has to be glidepath-api's own client, so a token minted for Clearance is refused here. Every other token still goes to TokenReview, so ServiceAccounts and Tower's identities work over the route too. The person becomes people:<their GitHub login>, and from there it's the same SubjectAccessReview as everyone else. Readers are bound by default; writing needs an explicit grant. Dex will sign in any GitHub user, so the API only lets in an explicit list, and the chart won't turn people sign-in on without one. A person is never a delegate.

I could have put the sign-in on the Gateway instead. Traefik doesn't do OIDC by itself, and more to the point, the API already had to decide every call. Two places that decide who you are is one more than I want to keep in agreement.

Tower can use the route as well, skipping the API server proxy, which should take a hop and some latency out of every call. That's merged, and switched off until I turn it on for prod. One thing to say plainly: the lab's Gateway is plain HTTP, so a token on that route crosses my home network unencrypted. It's the same for Clearance and Dex today. On a real network, TLS on the Gateway comes first.

And for agents

The API is plain HTTP on purpose. The agent side lives in Clearance, which puts MCP tools in front of it and maps the API's tiers onto its own. Nine new read tools landed this week, among them a bounded watch of a live run and the preview list. On October 29 a seeded typo broke a gate-api build. The event came off the stream, a triage agent read the run, its log and its explanation through the API, and it opened a one-line fix as a pull request, with nobody asking it to. How Clearance decides what an agent may do is its own story, and the next essay.

The rest of the week

A few smaller things landed in Glidepath alongside the API. Every app repo now runs the cicd-validate check on any pull request that changes cicd.yaml. A bare /retest comment now runs every check again: the check templates matched on a field a comment webhook doesn't have, so they were quietly skipped. Repos with nothing to build can be onboarded for pull-request checks alone. Re-run, cancel, promote and rollback have a run handle Tower can follow from the click to the pull request. Preview environments can be read through the API, though they still start with a preview label on the pull request. And the fleet lists page and send ETags, so an unchanged answer costs a 304.

What it costs, and what's still open

An API is one more thing to run, and I'd rather list its bill than have you find it.

• Prod depends on dev, more visibly. Tower on the prod cluster now gets every CI/CD answer from a service on the dev cluster. That dependency was already the top finding in the architecture review. The API makes it explicit; it doesn't remove it.
• Latency has a floor. Every call crosses between clusters, and under host load a cold read still gets close to Tower's ten seconds.
• Events aren't durable. Each replica keeps the last 2000 events, for a day at most, in memory. If both restart at once, a consumer that was away misses what happened. A design for durable fan-out over AMQP is written and deliberately deferred until a second unattended consumer appears, or a missed event costs something. It's known gap 44.
• The delegate is trusted. The API records the person Tower acts for but can't check them itself.
• The route is plain HTTP in the lab, as above.
• Some verbs stay in Tower. Ground promotions and cloud pins need a GitHub write identity the API doesn't have, and previews are read only.

A boring backend

My favourite result of all this is how dull Tower's backend got. It doesn't verify signatures, read Tekton Results, talk to Testkube or write PipelineRuns any more. It asks, and shows the answer. Tomorrow a different portal, a command-line tool or an agent can ask the same questions and get the same answers, decided by the same rules and written to the same log.

The third of my principles is an API-driven control plane, ready for AI. For Airframe that meant Kubernetes resources. For Glidepath it meant something smaller and plainer: an HTTP API that borrows the cluster's identity and RBAC, and keeps the verbs where people can find them.

James

← All architecture essays