Tower · Spec · Short-lived cluster credentials

No long-lived Kubernetes credential for Tower, anywhere

Tower reached the dev cluster with a ServiceAccount token that never expired, copied into four places. I wanted it gone, not rotated. The spec ended up doing more than that: checking what the token could really do turned up a path to far more power than a "read-only" identity should have, and that had to close first.

Spec October 10, 2026Status phases 0 to 4 done and live, Oct 12, 2026; phase 5 not started; Tower's Tekton and proxy grants cut back by Glidepath API P6, Oct 30, 2026Docs Tower cluster credentials
Phases 0 to 4 live · Oct 12, 2026

Tower now holds no long-lived Kubernetes credential.

What checking the token turned up

Finding 0.1

"Read-only" wasn't

The token could create PipelineRuns in any namespace, and nothing constrained their shape. Chained with a privileged ServiceAccount elsewhere, that was a path to cluster-admin. Worked out from RBAC and config, never exercised.

Finding 0.2

Anyone signed in could use it

Re-run and Cancel went from the browser through Backstage's generic Kubernetes proxy, which every signed-in user could reach. That bypassed Tower's ownership check entirely.

Phase 0

Closed first

An admission policy pins the exact PipelineRun shape Tower sends. Re-run and Cancel moved to owner-gated backend routes, and the proxy now refuses anything but reads.

Since Glidepath API P6 (Oct 30, 2026) tower-actor can't write a PipelineRun at all. Its binding to backstage-tekton-rerun is deleted, and the shape policy, renamed glidepath-api-pipelinerun-shape, refuses every PipelineRun write from it and holds glidepath-api, which now starts and cancels runs, to the same shape. tower-actor keeps view, the break-glass binder and the service proxy to glidepath-api. The reader lost its Rekor, Testkube and tekton-results-relay service-proxy grants; its Role backstage-ingestor-tekton-results-proxy keeps the old name but names only glidepath-api. The tokens themselves didn't change.

From a stored token to a federated one

From one stored token to federated tokensBefore, a never-expiring token lived in the dev cluster, a secrets store, the prod cluster and Tower's environment. After, the prod cluster signs short-lived tokens, Tower reads one from its pod, and the dev cluster trusts the prod issuer with rules on namespace, identity and lifetime.BEFORE · ONE TOKEN THAT NEVER EXPIRES, IN FOUR PLACESDev clustertoken SecretSecrets storeplanted by handProd clustersynced SecretTowerprocess envAFTER · NOTHING STORED, NOTHING THAT LIVES PAST TEN MINUTESProd issuerpublic keys onlyTower podtoken from kubeletDev clustertrusts the issuerSIGNS10 MIN · AUDIENCE DEVCEL rules: namespace, two names, lifetime
Figure 1The same pattern a managed cloud uses for workload identity, built by hand on two self-managed clusters.

Options considered

OptionVerdictWhy
Native federationChosenNothing stored, nothing new in the request path, the trust file reloads by itself, and claims are checked with CEL.
Secrets-store dynamic secretFallbackShort leases, but the store keeps a standing credential, and every Tower read depends on it being up.
Pinniped ConciergeFallbackSame trust model, plus a component with CA-key-level privilege. For clusters whose API server flags can't change.
A token "minter"RejectedIts bootstrap credential can mint the full-power token, so the risk stays the same.
Short-lived client certificatesRejectedSubmitting a request needs a credential first, and certificates can't be revoked.

It has to work beyond a home lab

The trust model is portable; the mechanism isn't. Managed control planes don't expose API server flags, so Tower picks how it gets a credential cluster by cluster. On the big clouds it's an exchange: the same projected token, traded for a cloud identity with a trust policy on its subject and audience. The reader and writer split carries over as two roles or two federated credentials. What gets coarser is the claim checking, and revocation is slower behind an exchange, because tokens already issued live until they expire.

The phases

Six phasesPhase 0 closed the hole, then TLS, federated reads, a split writer identity and deleting the old token. Phase 5, per-user writes, is not started.Close the holephase 0Verify TLSphase 1Federate readsphase 2Split writerphase 3Delete tokenphase 4Per-userphase 5Phases 0 to 4 built and live on Oct 12, 2026.Phase 5 carries each person's own identity into a write, so RBAC applies per team. It's its own spec.
Figure 2Phase 0 didn't depend on the rest and closed the larger risk, so it shipped first.

One regression on the way, about 21 minutes long: a Backstage proxy quirk sent dev-cluster reads to the prod cluster once the stored token was removed. The local harness never runs in a cluster, so it couldn't have caught it. A small guard fixed it, and a cleaner fix is noted for later.